Sep 11 16:17:04.931835 ip-10-0-141-84 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:17:05.296574 ip-10-0-141-84 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:05.296574 ip-10-0-141-84 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:17:05.296574 ip-10-0-141-84 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:05.296574 ip-10-0-141-84 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:17:05.296574 ip-10-0-141-84 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:17:05.298904 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.298824    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:17:05.303292 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303277    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:05.303292 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303293    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303298    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303302    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303305    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303308    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303311    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303314    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303316    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303319    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303322    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303325    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303328    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303330    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303333    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303336    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303338    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303341    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303343    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303346    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303348    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:05.303356 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303352    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303354    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303357    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303360    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303363    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303366    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303368    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303371    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303374    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303377    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303379    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303382    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303384    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303387    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303389    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303392    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303394    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303397    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303399    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303402    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:05.303863 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303404    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303407    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303409    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303412    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303414    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303416    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303419    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303421    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303424    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303426    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303429    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303431    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303434    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303437    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303440    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303448    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303451    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303453    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303456    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303458    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:05.304517 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303461    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303463    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303466    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303468    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303471    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303473    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303476    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303480    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303483    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303486    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303489    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303492    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303494    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303497    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303499    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303502    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303504    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303507    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303510    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:05.305252 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303513    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303515    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303519    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303522    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303525    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303529    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303532    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303536    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303553    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303556    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303559    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303562    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303564    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303567    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303570    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303572    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303575    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303577    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.303580    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:05.305746 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305558    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305573    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305579    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305584    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305587    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305590    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305594    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305598    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305600    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305603    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305606    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305609    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305611    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305614    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305616    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305619    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305621    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305624    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305626    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:05.306244 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305629    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305632    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305635    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305638    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305641    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305643    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305646    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305649    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305652    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305654    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305657    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305660    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305663    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305681    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305686    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305689    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305692    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305695    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305698    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305701    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:05.306741 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305703    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305706    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305708    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305711    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305713    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305716    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305720    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305724    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305727    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305730    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305733    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305742    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305745    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305748    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305751    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305753    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305756    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305758    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305761    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:05.307265 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305764    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305767    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305770    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305772    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305775    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305778    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305782    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305786    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305789    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305792    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305794    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305797    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305800    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305802    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305805    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305807    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305810    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305812    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305815    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305818    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:05.307747 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305820    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305822    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305825    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305827    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305830    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305838    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305841    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305843    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305846    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305848    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305851    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305853    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305856    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305859    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305862    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305865    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305867    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305869    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305872    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305875    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:05.308243 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.305877    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306264    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306268    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306271    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306274    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306276    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306279    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306282    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306284    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306289    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306293    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306296    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306299    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306302    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306305    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306309    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306312    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306322    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306325    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:05.308745 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306328    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306331    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306334    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306337    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306340    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306343    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306346    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306348    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306352    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306354    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306357    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306360    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306363    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306366    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306368    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306371    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306374    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306376    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306378    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306381    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:05.309212 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306384    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306386    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306388    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306391    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306393    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306396    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306399    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306401    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306403    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306406    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306409    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306417    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306420    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306422    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306425    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306427    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306430    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306433    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306435    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306438    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:05.309739 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306441    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306443    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306445    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306448    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306451    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306453    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306456    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306458    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306462    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306465    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306469    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306472    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306474    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306477    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306479    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306482    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306484    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306487    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306490    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306493    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:05.310217 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306495    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306498    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306500    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306502    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306511    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306514    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306516    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306519    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306522    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306524    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306527    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306530    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306532    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306535    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306554    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306558    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306561    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306563    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306566    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306568    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:05.310726 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.306571    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306638    2550 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306672    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306681    2550 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306685    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306689    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306692    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306700    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306708    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306711    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306714    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306718    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306721    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306724    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306727    2550 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306730    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306732    2550 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306741    2550 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306744    2550 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306750    2550 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306753    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306756    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306759    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306762    2550 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:17:05.311201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306766    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306768    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306772    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306775    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306778    2550 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306781    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306784    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306787    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306790    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306794    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306796    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306799    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306802    2550 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306805    2550 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306808    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306815    2550 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306818    2550 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306821    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306824    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306827    2550 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306830    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306833    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306836    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306839    2550 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306842    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:17:05.311796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306844    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306853    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306856    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306859    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306861    2550 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306864    2550 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306868    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306871    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306873    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306877    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306880    2550 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306883    2550 flags.go:64] FLAG: --help="false"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306886    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306889    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306891    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306894    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306897    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306900    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306903    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306906    2550 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306909    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306912    2550 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306915    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306918    2550 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:17:05.312443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306921    2550 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306924    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306926    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306929    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306932    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306935    2550 flags.go:64] FLAG: --lock-file=""
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306937    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306940    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306943    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306948    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306957    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306960    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306962    2550 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306966    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306969    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306972    2550 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306975    2550 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306979    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306982    2550 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306986    2550 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306988    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306991    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306994    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.306997    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307000    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:17:05.313062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307003    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307005    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307012    2550 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307015    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307018    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307021    2550 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307024    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307029    2550 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307031    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307034    2550 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307037    2550 flags.go:64] FLAG: --port="10250"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307040    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307043    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0340f9eca3f7cadeb"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307046    2550 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307048    2550 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307051    2550 flags.go:64] FLAG: --register-node="true"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307054    2550 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307057    2550 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307065    2550 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307068    2550 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307071    2550 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307074    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307077    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307080    2550 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307083    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:17:05.313747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307086    2550 flags.go:64] FLAG: --runonce="false"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307089    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307091    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307094    2550 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307097    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307100    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307103    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307106    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307109    2550 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307112    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307115    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307118    2550 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307120    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307123    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307126    2550 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307129    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307134    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307137    2550 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307140    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307147    2550 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307150    2550 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307153    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307155    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307158    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307161    2550 flags.go:64] FLAG: --v="2"
Sep 11 16:17:05.314367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307164    2550 flags.go:64] FLAG: --version="false"
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307173    2550 flags.go:64] FLAG: --vmodule=""
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307177    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.307180    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307313    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307317    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307320    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307323    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307326    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307329    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307332    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307335    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307338    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307341    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307344    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307347    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307350    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307354    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307356    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:05.314979 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307361    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307365    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307368    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307371    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307374    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307376    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307379    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307382    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307384    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307387    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307390    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307392    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307395    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307398    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307400    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307408    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307412    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307416    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307419    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307421    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:05.315455 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307424    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307427    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307430    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307432    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307435    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307438    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307440    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307443    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307454    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307457    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307461    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307463    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307466    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307469    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307472    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307474    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307477    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307479    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307482    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307485    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:05.315994 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307487    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307490    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307493    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307495    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307498    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307501    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307503    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307506    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307514    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307519    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307521    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307523    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307526    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307529    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307531    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307534    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307536    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307552    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307557    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:05.316473 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307560    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307566    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307571    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307577    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307582    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307586    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307590    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307593    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307596    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307599    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307602    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307605    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307608    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307610    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307613    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307616    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307619    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307621    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307624    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307627    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:05.316955 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307630    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307632    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307642    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307645    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.307648    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.308296    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.314937    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.314951    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.314972    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.314977    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315056    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315061    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315064    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315067    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315071    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:05.317461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315073    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315077    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315080    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315083    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315085    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315088    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315090    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315093    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315096    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315098    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315101    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315104    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315106    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315110    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315114    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315117    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315121    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315125    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315132    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:05.317890 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315138    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315143    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315147    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315151    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315156    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315159    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315162    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315165    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315169    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315172    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315175    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315178    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315181    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315183    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315186    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315188    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315191    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315194    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315196    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315199    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:05.318388 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315201    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315204    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315206    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315209    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315211    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315214    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315216    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315219    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315222    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315225    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315228    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315231    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315233    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315237    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315240    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315242    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315245    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315248    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315250    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:05.318972 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315253    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315255    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315258    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315260    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315263    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315265    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315268    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315270    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315273    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315275    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315278    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315281    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315283    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315286    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315288    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315291    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315294    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315296    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315299    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315301    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:05.319448 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315304    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315306    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315309    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315311    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315314    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315316    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315319    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315322    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315325    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315327    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315330    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315332    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315335    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315337    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315340    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:05.320186 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315342    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.315347    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315467    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315471    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315475    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315478    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315481    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315484    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315487    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315490    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315492    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315495    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315497    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315500    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315502    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:17:05.320584 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315505    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315507    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315510    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315513    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315515    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315518    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315520    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315522    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315525    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315527    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315530    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315533    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315536    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315538    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315553    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315556    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315559    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315562    2550 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315564    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315566    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315569    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:17:05.321105 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315572    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315574    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315577    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315579    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315582    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315585    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315587    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315590    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315593    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315595    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315599    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315603    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315606    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315608    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315611    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315614    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315616    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315619    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315621    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:17:05.321866 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315624    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315627    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315629    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315632    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315635    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315638    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315640    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315643    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315646    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315648    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315651    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315653    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315656    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315659    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315661    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315664    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315667    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315669    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315672    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315674    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:17:05.322461 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315677    2550 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315679    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315682    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315684    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315687    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315689    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315692    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315694    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315697    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315699    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315703    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315706    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315709    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315712    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315714    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315717    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315720    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315723    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315725    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:17:05.323029 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315728    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315732    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315735    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315738    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315740    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315743    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:05.315746    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.315751    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.315841    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.317680    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.318716    2550 server.go:1019] "Starting client certificate rotation"
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.318821    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.318875    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:17:05.323573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.321144    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:17:05.323903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.322482    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:17:05.335928 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.335908    2550 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:17:05.340848 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.340831    2550 log.go:25] "Validated CRI v1 image API"
Sep 11 16:17:05.342597 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.342512    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:17:05.345753 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.345736    2550 fs.go:135] Filesystem UUIDs: map[220d941a-2f8e-4df5-addf-f10b8b365df8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e87f1cbe-202a-4edc-8a6a-9aeaa3a6ccda:/dev/nvme0n1p3]
Sep 11 16:17:05.345835 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.345753    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:17:05.348383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.348366    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:17:05.350743 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.350643    2550 manager.go:216] Machine: {Timestamp:2026-09-11 16:17:05.349265976 +0000 UTC m=+0.305313595 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100611 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2da8f8cae52aca8f1bc7355c9ff577 SystemUUID:ec2da8f8-cae5-2aca-8f1b-c7355c9ff577 BootID:87c82848-7f70-4635-8244-6fea5ab10811 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:bd:8f:e5:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:bd:8f:e5:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:07:9a:39:34:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:17:05.350743 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.350738    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:17:05.350859 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.350809    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:17:05.353951 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.353922    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:17:05.354108 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.353956    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-84.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:17:05.355634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.355622    2550 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:17:05.355690 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.355640    2550 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:17:05.355690 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.355652    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:17:05.356319 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.356308    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:17:05.357154 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.357141    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:17:05.360811 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360799    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:17:05.360931 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360922    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:17:05.360970 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360951    2550 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:17:05.360970 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360963    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:17:05.361027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360978    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:17:05.361027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360988    2550 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:17:05.361027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.360997    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:17:05.361323 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.361311    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:17:05.361359 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.361344    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:17:05.364166 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.364153    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:17:05.366337 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.366324    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:17:05.366374 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.366354    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:17:05.367746 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367735    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:17:05.367779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367751    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:17:05.367779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367758    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:17:05.367779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367763    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:17:05.367779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367772    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:17:05.367779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367779    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367792    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367797    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367803    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367808    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367816    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367822    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367845    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:17:05.367905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.367850    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:17:05.371588 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.371576    2550 server.go:1268] "Started kubelet"
Sep 11 16:17:05.371668 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.371643    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:17:05.371804 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.371734    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:17:05.371901 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.371829    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:17:05.372450 ip-10-0-141-84 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:17:05.372876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.372811    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:17:05.372876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.373019    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-84.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:17:05.373618 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.373595    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:17:05.373671 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.373595    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:17:05.374353 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.374340    2550 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:17:05.376018 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.375994    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p4hls"
Sep 11 16:17:05.376670 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.375730    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-84.ec2.internal.18d45077517eee40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-84.ec2.internal,UID:ip-10-0-141-84.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-84.ec2.internal,},FirstTimestamp:2026-09-11 16:17:05.371532864 +0000 UTC m=+0.327580484,LastTimestamp:2026-09-11 16:17:05.371532864 +0000 UTC m=+0.327580484,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-84.ec2.internal,}"
Sep 11 16:17:05.379016 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.378993    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:17:05.379185 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.379153    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:17:05.379504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.379489    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:17:05.379673 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.379653    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.380132 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380114    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:17:05.380204 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380135    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:17:05.380204 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380114    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:17:05.380303 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380208    2550 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:17:05.380303 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380216    2550 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:17:05.380757 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380734    2550 factory.go:55] Registering systemd factory
Sep 11 16:17:05.380844 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.380769    2550 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:17:05.381021 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.381005    2550 factory.go:153] Registering CRI-O factory
Sep 11 16:17:05.381067 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.381026    2550 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:17:05.381123 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.381083    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:17:05.381123 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.381102    2550 factory.go:103] Registering Raw factory
Sep 11 16:17:05.381123 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.381117    2550 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:17:05.381255 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.381115    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 16:17:05.381255 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.381224    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-84.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 16:17:05.381440 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.381428    2550 manager.go:349] Starting recovery of all containers
Sep 11 16:17:05.382207 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.382189    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p4hls"
Sep 11 16:17:05.393228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.393118    2550 manager.go:354] Recovery completed
Sep 11 16:17:05.397437 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.397422    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.399399 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.399384    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.399474 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.399410    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.399474 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.399420    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.400256 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.400044    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:17:05.400312 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.400263    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:17:05.400362 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.400044    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.400893 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.400480    2550 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:17:05.401636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.401620    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.401712 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.401649    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.401712 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.401659    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.402125 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.402112    2550 policy_none.go:49] "None policy: Start"
Sep 11 16:17:05.402191 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.402128    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:17:05.402191 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.402139    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:17:05.403042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.403031    2550 policy_none.go:47] "Start"
Sep 11 16:17:05.436116 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436102    2550 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.436134    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436143    2550 server.go:87] "Starting device plugin registration server"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436424    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436434    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436615    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436681    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.436690    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.437116    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:17:05.441213 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.437152    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.469835 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.469810    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:17:05.470904 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.470888    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:17:05.470972 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.470932    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:17:05.470972 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.470952    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:17:05.471074 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.470984    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 16:17:05.473909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.473891    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:05.537247 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.537223    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.538156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.538139    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.538254 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.538166    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.538254 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.538176    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.538254 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.538206    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.546872 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.546808    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.546872 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.546833    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-84.ec2.internal\": node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.566978 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.566953    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.571061 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.571047    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"]
Sep 11 16:17:05.571112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.571101    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.571865 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.571852    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.571932 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.571878    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.571932 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.571887    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.573359 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.573348    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.573496 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.573483    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.573531 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.573512    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.574462 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.574446    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.574462 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.574459    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.574632 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.574478    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.574632 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.574494    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.574632 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.574480    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.574632 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.574566    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.575695 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.575680    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.575760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.575710    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:17:05.576518 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.576502    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:17:05.576614 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.576532    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:17:05.576614 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.576558    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:17:05.582108 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.582080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a048dfbc799f86a2baabe7fe5ea46cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal\" (UID: \"4a048dfbc799f86a2baabe7fe5ea46cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.582185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.582118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a048dfbc799f86a2baabe7fe5ea46cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal\" (UID: \"4a048dfbc799f86a2baabe7fe5ea46cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.582185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.582143    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/520fabedc60a64e4fd6648fb59243517-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.ec2.internal\" (UID: \"520fabedc60a64e4fd6648fb59243517\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.592055 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.592040    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-84.ec2.internal\" not found" node="ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.596234 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.596220    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-84.ec2.internal\" not found" node="ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.668015 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.667997    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.682442 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.682423    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a048dfbc799f86a2baabe7fe5ea46cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal\" (UID: \"4a048dfbc799f86a2baabe7fe5ea46cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.682514 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.682446    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a048dfbc799f86a2baabe7fe5ea46cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal\" (UID: \"4a048dfbc799f86a2baabe7fe5ea46cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.682514 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.682465    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/520fabedc60a64e4fd6648fb59243517-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.ec2.internal\" (UID: \"520fabedc60a64e4fd6648fb59243517\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.682596 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.682527    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/520fabedc60a64e4fd6648fb59243517-config\") pod \"kube-apiserver-proxy-ip-10-0-141-84.ec2.internal\" (UID: \"520fabedc60a64e4fd6648fb59243517\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.682596 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.682534    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a048dfbc799f86a2baabe7fe5ea46cb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal\" (UID: \"4a048dfbc799f86a2baabe7fe5ea46cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.682596 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.682538    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a048dfbc799f86a2baabe7fe5ea46cb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal\" (UID: \"4a048dfbc799f86a2baabe7fe5ea46cb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.768705 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.768683    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.869458 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.869395    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:05.894826 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.894808    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.899634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:05.899619    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:05.970305 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:05.970279    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.070752 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.070730    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.171302 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.171241    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.271860 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.271828    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.318517 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.318480    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:17:06.319030 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.318655    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:17:06.372330 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.372305    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.386557 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.386499    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:12:05 +0000 UTC" deadline="2028-03-19 12:07:00.044464132 +0000 UTC"
Sep 11 16:17:06.386557 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.386539    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13315h49m53.657930074s"
Sep 11 16:17:06.472517 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.472497    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.478965 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:06.478938    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a048dfbc799f86a2baabe7fe5ea46cb.slice/crio-d27c0052bccf17e263cf2a760f166713a6199b307003129fce5cf2992cba791d WatchSource:0}: Error finding container d27c0052bccf17e263cf2a760f166713a6199b307003129fce5cf2992cba791d: Status 404 returned error can't find the container with id d27c0052bccf17e263cf2a760f166713a6199b307003129fce5cf2992cba791d
Sep 11 16:17:06.479170 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:06.479154    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520fabedc60a64e4fd6648fb59243517.slice/crio-24352fa04d14c59402ee6beff12bb8fcba02f51634f5d31ea4bc19c309af3be4 WatchSource:0}: Error finding container 24352fa04d14c59402ee6beff12bb8fcba02f51634f5d31ea4bc19c309af3be4: Status 404 returned error can't find the container with id 24352fa04d14c59402ee6beff12bb8fcba02f51634f5d31ea4bc19c309af3be4
Sep 11 16:17:06.484225 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.484209    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:17:06.520100 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.520081    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:06.576862 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.576841    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.680225 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.680196    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.782284 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.782222    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.809585 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.809567    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:06.882715 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:06.882689    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-84.ec2.internal\" not found"
Sep 11 16:17:06.883524 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.883509    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:06.979752 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.979726    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:06.994042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.994021    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:17:06.995180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:06.995161    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal"
Sep 11 16:17:07.001894 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.001873    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:17:07.362825 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.362797    2550 apiserver.go:52] "Watching apiserver"
Sep 11 16:17:07.371832 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.371784    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:17:07.372696 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.372664    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-vcxfz","openshift-ovn-kubernetes/ovnkube-node-s6jwt","kube-system/global-pull-secret-syncer-754qk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d","openshift-image-registry/node-ca-9wbwf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal","openshift-multus/multus-additional-cni-plugins-dlccg","openshift-multus/multus-qk8cl","openshift-network-diagnostics/network-check-target-bxrwt","openshift-network-operator/iptables-alerter-xxs9n","kube-system/konnectivity-agent-vg278","kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lwqsr","openshift-dns/node-resolver-jx4cp"]
Sep 11 16:17:07.374768 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.374746    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.376584 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.376361    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.378184 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.377637    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.378184 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.377721    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:07.378184 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.377985    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:17:07.378184 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.377989    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.378365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.378213    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.378615 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.378430    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:17:07.378615 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.378573    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5h2pr\""
Sep 11 16:17:07.379246 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.378805    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.379246 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.379054    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.379790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.379489    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kjtcq\""
Sep 11 16:17:07.379790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.379573    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.380824 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.380168    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:17:07.380824 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.380296    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:17:07.381356 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.381211    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.381826 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.381648    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:17:07.381826 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.381725    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sqgqx\""
Sep 11 16:17:07.381826 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.381733    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.382415 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.382186    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.383677 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.383403    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.383677 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.383629    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:17:07.384134 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.384110    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.384413 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.384392    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xmllf\""
Sep 11 16:17:07.384592 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.384529    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.384955 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.384937    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:07.385080 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.385053    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:07.386504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.386411    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.386755 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.386646    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.386940 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.386909    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qh2x\""
Sep 11 16:17:07.387434 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.387415    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:07.387530 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.387496    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:07.388237 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.388219    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.390020 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.389998    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:17:07.390501 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.390430    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.391297 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.391274    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:17:07.391520 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.391489    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:17:07.391783 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.391764    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.392011 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.391993    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hxbmp\""
Sep 11 16:17:07.392249 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392216    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:17:07.392407 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392384    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.392521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392420    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3425dd2f-36b9-4a54-bdcb-1989ae091941-host\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.392521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392456    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cni-binary-copy\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.392521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392481    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysconfig\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392505    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-lib-modules\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392528    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysctl-d\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392550    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3425dd2f-36b9-4a54-bdcb-1989ae091941-serviceca\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-systemd\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392649    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-run\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392674    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/231ac950-39f0-4afb-a966-3414a5124d8b-kubelet-config\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392698    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-registration-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ee5227-cc0b-4691-a0be-4334a05612dc-tmp\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392795    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392828    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-host\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.392877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392859    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfdn\" (UniqueName: \"kubernetes.io/projected/d5ee5227-cc0b-4691-a0be-4334a05612dc-kube-api-access-nnfdn\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-device-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392914    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392939    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-modprobe-d\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392967    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/231ac950-39f0-4afb-a966-3414a5124d8b-dbus\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.392991    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-socket-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393051    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-sys-fs\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393077    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8j2\" (UniqueName: \"kubernetes.io/projected/3425dd2f-36b9-4a54-bdcb-1989ae091941-kube-api-access-hr8j2\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393115    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-sys\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393138    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-tuned\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393167    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6gk\" (UniqueName: \"kubernetes.io/projected/9341296d-c6a8-495b-8413-1758fbc4d091-kube-api-access-md6gk\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393192    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-system-cni-dir\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393218    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99zl\" (UniqueName: \"kubernetes.io/projected/b1a4f2f1-7987-46fa-ae27-600b250b0d12-kube-api-access-x99zl\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-kubernetes\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393275    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-var-lib-kubelet\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.393537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393303    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cnibin\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.394437 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.393726    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.394437 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.394305    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:17:07.394551 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.394453    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.394729 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.394692    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:17:07.394827 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.394778    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:17:07.395066 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.395046    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q6jbx\""
Sep 11 16:17:07.395187 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.395166    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.395300 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.395278    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:17:07.396035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.396005    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysctl-conf\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.398229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.397545    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:17:07.398229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.397830    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gcbzg\""
Sep 11 16:17:07.398229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.397839    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:17:07.398229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.398076    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9r49b\""
Sep 11 16:17:07.398229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.398169    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:17:07.422794 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.422762    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nnp22"
Sep 11 16:17:07.435074 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.435053    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nnp22"
Sep 11 16:17:07.446220 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.446200    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:07.477289 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.477241    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal" event={"ID":"4a048dfbc799f86a2baabe7fe5ea46cb","Type":"ContainerStarted","Data":"d27c0052bccf17e263cf2a760f166713a6199b307003129fce5cf2992cba791d"}
Sep 11 16:17:07.478599 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.478575    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal" event={"ID":"520fabedc60a64e4fd6648fb59243517","Type":"ContainerStarted","Data":"24352fa04d14c59402ee6beff12bb8fcba02f51634f5d31ea4bc19c309af3be4"}
Sep 11 16:17:07.482086 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.482063    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:17:07.496745 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496719    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysctl-d\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.496851 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496759    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.496851 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496783    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:07.496851 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496809    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmkxw\" (UniqueName: \"kubernetes.io/projected/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-kube-api-access-hmkxw\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:07.496851 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496843    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-ovnkube-config\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496865    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-cni-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496887    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-etc-kubernetes\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/231ac950-39f0-4afb-a966-3414a5124d8b-kubelet-config\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496936    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-registration-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496950    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysctl-d\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496975    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.496961    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497057    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-registration-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497061    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-etc-selinux\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497065    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/231ac950-39f0-4afb-a966-3414a5124d8b-kubelet-config\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497091    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ee5227-cc0b-4691-a0be-4334a05612dc-tmp\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497119    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-systemd\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497144    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-ovn\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497169    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-ovnkube-script-lib\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497198    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497226    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfdn\" (UniqueName: \"kubernetes.io/projected/d5ee5227-cc0b-4691-a0be-4334a05612dc-kube-api-access-nnfdn\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5q9\" (UniqueName: \"kubernetes.io/projected/92ffbbfc-30e6-4dfe-815d-713685253804-kube-api-access-kw5q9\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497273    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-conf-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497300    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.497340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497326    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pknc\" (UniqueName: \"kubernetes.io/projected/ca53fdc1-656c-4a18-b255-78646e6fb4f9-kube-api-access-9pknc\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497350    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-run-netns\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497372    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-cnibin\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497393    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-socket-dir-parent\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497414    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-kubelet\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497435    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497456    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-socket-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497476    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-sys-fs\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497498    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-sys\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-tuned\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6gk\" (UniqueName: \"kubernetes.io/projected/9341296d-c6a8-495b-8413-1758fbc4d091-kube-api-access-md6gk\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497575    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99zl\" (UniqueName: \"kubernetes.io/projected/b1a4f2f1-7987-46fa-ae27-600b250b0d12-kube-api-access-x99zl\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497602    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-kubernetes\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497637    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-var-lib-kubelet\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497658    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-cni-bin\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497686    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca-konnectivity-ca\") pod \"konnectivity-agent-vg278\" (UID: \"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca\") " pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-hostroot\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.497819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497743    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cnibin\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497780    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysctl-conf\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497803    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497837    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca-agent-certs\") pod \"konnectivity-agent-vg278\" (UID: \"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca\") " pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497852    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497887    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3306db7f-f969-4cac-a413-5a9400e48e99-multus-daemon-config\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497918    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-multus-certs\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5f76\" (UniqueName: \"kubernetes.io/projected/3306db7f-f969-4cac-a413-5a9400e48e99-kube-api-access-k5f76\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497972    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498025    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3425dd2f-36b9-4a54-bdcb-1989ae091941-host\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498054    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-lib-modules\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498056    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cnibin\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498165    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysctl-conf\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498218    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-lib-modules\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.497890    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498235    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-var-lib-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498341    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-log-socket\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.498600 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498355    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3425dd2f-36b9-4a54-bdcb-1989ae091941-host\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498372    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-k8s-cni-cncf-io\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498403    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3425dd2f-36b9-4a54-bdcb-1989ae091941-serviceca\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-systemd\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498462    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-run\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.498486    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498493    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b902a3c9-3153-4972-a0d5-eff6e6aec791-tmp-dir\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498519    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b902a3c9-3153-4972-a0d5-eff6e6aec791-hosts-file\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.498556    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.998533245 +0000 UTC m=+2.954580866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498581    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3306db7f-f969-4cac-a413-5a9400e48e99-cni-binary-copy\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498631    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-host\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498666    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-socket-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498691    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-host\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498719    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-sys-fs\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.498763    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-sys\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499154    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499201    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-var-lib-kubelet\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.499489 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499205    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvqz\" (UniqueName: \"kubernetes.io/projected/b902a3c9-3153-4972-a0d5-eff6e6aec791-kube-api-access-nxvqz\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-cni-multus\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499250    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499266    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-systemd\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499276    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-device-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499327    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9341296d-c6a8-495b-8413-1758fbc4d091-device-dir\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499327    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-run\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499341    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-modprobe-d\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499378    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-kubernetes\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499380    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-systemd-units\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499429    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-slash\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499432    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-modprobe-d\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499474    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92ffbbfc-30e6-4dfe-815d-713685253804-ovn-node-metrics-cert\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-netns\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499543    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca53fdc1-656c-4a18-b255-78646e6fb4f9-host-slash\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499590    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/231ac950-39f0-4afb-a966-3414a5124d8b-dbus\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499617    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8j2\" (UniqueName: \"kubernetes.io/projected/3425dd2f-36b9-4a54-bdcb-1989ae091941-kube-api-access-hr8j2\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.500681 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499665    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-cni-netd\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499691    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-env-overrides\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499720    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-system-cni-dir\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499744    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca53fdc1-656c-4a18-b255-78646e6fb4f9-iptables-alerter-script\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499810    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/231ac950-39f0-4afb-a966-3414a5124d8b-dbus\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499827    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499854    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a4f2f1-7987-46fa-ae27-600b250b0d12-system-cni-dir\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499851    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3425dd2f-36b9-4a54-bdcb-1989ae091941-serviceca\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499884    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-node-log\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.499919    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500054    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-cni-bin\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500087    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-kubelet\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500114    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500137    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-system-cni-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500166    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cni-binary-copy\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500193    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysconfig\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500215    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-etc-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.503140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500282    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-sysconfig\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.503830 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.500672    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a4f2f1-7987-46fa-ae27-600b250b0d12-cni-binary-copy\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.503830 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.503134    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5ee5227-cc0b-4691-a0be-4334a05612dc-etc-tuned\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.504168 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.504136    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5ee5227-cc0b-4691-a0be-4334a05612dc-tmp\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.517779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.517371    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfdn\" (UniqueName: \"kubernetes.io/projected/d5ee5227-cc0b-4691-a0be-4334a05612dc-kube-api-access-nnfdn\") pod \"tuned-lwqsr\" (UID: \"d5ee5227-cc0b-4691-a0be-4334a05612dc\") " pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.517779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.517674    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6gk\" (UniqueName: \"kubernetes.io/projected/9341296d-c6a8-495b-8413-1758fbc4d091-kube-api-access-md6gk\") pod \"aws-ebs-csi-driver-node-bvc7d\" (UID: \"9341296d-c6a8-495b-8413-1758fbc4d091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.518138 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.518116    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99zl\" (UniqueName: \"kubernetes.io/projected/b1a4f2f1-7987-46fa-ae27-600b250b0d12-kube-api-access-x99zl\") pod \"multus-additional-cni-plugins-dlccg\" (UID: \"b1a4f2f1-7987-46fa-ae27-600b250b0d12\") " pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.519160 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.519134    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8j2\" (UniqueName: \"kubernetes.io/projected/3425dd2f-36b9-4a54-bdcb-1989ae091941-kube-api-access-hr8j2\") pod \"node-ca-9wbwf\" (UID: \"3425dd2f-36b9-4a54-bdcb-1989ae091941\") " pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.600721 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600687    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-run-netns\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.600871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600726    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-cnibin\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.600871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600752    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-socket-dir-parent\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.600871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600796    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-kubelet\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.600871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600824    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-cnibin\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.600871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600822    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-run-netns\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.600871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600837    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-cni-bin\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600883    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-kubelet\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600891    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-cni-bin\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600906    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca-konnectivity-ca\") pod \"konnectivity-agent-vg278\" (UID: \"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca\") " pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600935    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-hostroot\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600952    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-socket-dir-parent\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.600962    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601010    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601017    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca-agent-certs\") pod \"konnectivity-agent-vg278\" (UID: \"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca\") " pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.601200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601029    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-hostroot\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.601627 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601051    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3306db7f-f969-4cac-a413-5a9400e48e99-multus-daemon-config\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.601627 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601418    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-multus-certs\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.601737 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601468    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5f76\" (UniqueName: \"kubernetes.io/projected/3306db7f-f969-4cac-a413-5a9400e48e99-kube-api-access-k5f76\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.601737 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601730    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-var-lib-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601844 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-log-socket\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.601955 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601929    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-k8s-cni-cncf-io\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.602070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601967    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b902a3c9-3153-4972-a0d5-eff6e6aec791-tmp-dir\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.602181 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.601998    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b902a3c9-3153-4972-a0d5-eff6e6aec791-hosts-file\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.602255 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.602203    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3306db7f-f969-4cac-a413-5a9400e48e99-cni-binary-copy\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.602984 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.602957    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvqz\" (UniqueName: \"kubernetes.io/projected/b902a3c9-3153-4972-a0d5-eff6e6aec791-kube-api-access-nxvqz\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.603074 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603011    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-cni-multus\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603074 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603054    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-systemd-units\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603087    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-slash\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92ffbbfc-30e6-4dfe-815d-713685253804-ovn-node-metrics-cert\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603169    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-netns\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603326 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603228    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca53fdc1-656c-4a18-b255-78646e6fb4f9-host-slash\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.603326 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603264    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-cni-netd\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603326 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603294    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-env-overrides\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603476 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603416    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca53fdc1-656c-4a18-b255-78646e6fb4f9-iptables-alerter-script\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.603476 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603437    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca-konnectivity-ca\") pod \"konnectivity-agent-vg278\" (UID: \"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca\") " pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.603476 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:07.603637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603502    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-node-log\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603510    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-cni-multus\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603011    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-var-lib-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603547    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603556    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3306db7f-f969-4cac-a413-5a9400e48e99-multus-daemon-config\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603648    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-multus-certs\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603663    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-cni-bin\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603623    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-cni-bin\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603711    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-kubelet\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.603737    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603777    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-system-cni-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.603803    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.10378443 +0000 UTC m=+3.059832041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.603918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603844    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-etc-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603926    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-system-cni-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603938    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603971    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmkxw\" (UniqueName: \"kubernetes.io/projected/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-kube-api-access-hmkxw\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603986    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-systemd-units\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.603995    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-ovnkube-config\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604073    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-slash\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-cni-netd\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604192    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-cni-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604241    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-etc-kubernetes\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604281    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-systemd\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-ovn\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604345    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-ovnkube-script-lib\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604409    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-cni-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.604793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604383    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5q9\" (UniqueName: \"kubernetes.io/projected/92ffbbfc-30e6-4dfe-815d-713685253804-kube-api-access-kw5q9\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604496    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-node-log\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.604793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604520    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-conf-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.604793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.604560    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pknc\" (UniqueName: \"kubernetes.io/projected/ca53fdc1-656c-4a18-b255-78646e6fb4f9-kube-api-access-9pknc\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.605177 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.605143    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-etc-openvswitch\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.605428 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.605405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca53fdc1-656c-4a18-b255-78646e6fb4f9-iptables-alerter-script\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.605562 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.605514    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-var-lib-kubelet\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.605678 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.605609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-host-run-ovn-kubernetes\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.606160 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606082    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-systemd\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.606259 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606167    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-netns\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.606259 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606223    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca53fdc1-656c-4a18-b255-78646e6fb4f9-host-slash\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.606544 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606487    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-ovnkube-config\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.606544 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606498    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-etc-kubernetes\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.606729 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606576    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-run-ovn\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.606729 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606682    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-env-overrides\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.606848 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606805    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-multus-conf-dir\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.606907 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.606874    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92ffbbfc-30e6-4dfe-815d-713685253804-log-socket\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.607152 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.607112    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b902a3c9-3153-4972-a0d5-eff6e6aec791-tmp-dir\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.607152 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.607127    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92ffbbfc-30e6-4dfe-815d-713685253804-ovnkube-script-lib\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.607301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.607204    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3306db7f-f969-4cac-a413-5a9400e48e99-host-run-k8s-cni-cncf-io\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.607301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.607275    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b902a3c9-3153-4972-a0d5-eff6e6aec791-hosts-file\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.607451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.607431    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3306db7f-f969-4cac-a413-5a9400e48e99-cni-binary-copy\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.609497 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.609459    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca-agent-certs\") pod \"konnectivity-agent-vg278\" (UID: \"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca\") " pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:07.611205 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.611147    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92ffbbfc-30e6-4dfe-815d-713685253804-ovn-node-metrics-cert\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.611575 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.611539    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:07.611684 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.611579    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:07.611684 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.611593    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.611684 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:07.611655    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:08.111637548 +0000 UTC m=+3.067685174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.613940 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.613881    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pknc\" (UniqueName: \"kubernetes.io/projected/ca53fdc1-656c-4a18-b255-78646e6fb4f9-kube-api-access-9pknc\") pod \"iptables-alerter-xxs9n\" (UID: \"ca53fdc1-656c-4a18-b255-78646e6fb4f9\") " pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.615131 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.615091    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5f76\" (UniqueName: \"kubernetes.io/projected/3306db7f-f969-4cac-a413-5a9400e48e99-kube-api-access-k5f76\") pod \"multus-qk8cl\" (UID: \"3306db7f-f969-4cac-a413-5a9400e48e99\") " pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.616121 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.616094    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmkxw\" (UniqueName: \"kubernetes.io/projected/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-kube-api-access-hmkxw\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:07.616300 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.616274    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5q9\" (UniqueName: \"kubernetes.io/projected/92ffbbfc-30e6-4dfe-815d-713685253804-kube-api-access-kw5q9\") pod \"ovnkube-node-s6jwt\" (UID: \"92ffbbfc-30e6-4dfe-815d-713685253804\") " pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.616771 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.616749    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvqz\" (UniqueName: \"kubernetes.io/projected/b902a3c9-3153-4972-a0d5-eff6e6aec791-kube-api-access-nxvqz\") pod \"node-resolver-jx4cp\" (UID: \"b902a3c9-3153-4972-a0d5-eff6e6aec791\") " pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.696401 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.696365    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dlccg"
Sep 11 16:17:07.704379 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.704353    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lwqsr"
Sep 11 16:17:07.717339 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.717316    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qk8cl"
Sep 11 16:17:07.724249 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.724225    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d"
Sep 11 16:17:07.733962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.733941    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wbwf"
Sep 11 16:17:07.742568 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.742547    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jx4cp"
Sep 11 16:17:07.752140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.752114    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:07.762035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.762015    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xxs9n"
Sep 11 16:17:07.766756 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:07.766737    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:08.008404 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:08.008311    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:08.008582 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.008472    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:08.008582 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.008541    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:09.008523805 +0000 UTC m=+3.964571431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:08.109093 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:08.109055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:08.109244 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.109187    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:08.109295 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.109253    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:09.10923766 +0000 UTC m=+4.065285279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:08.209712 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:08.209676    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:08.209870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.209806    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:08.209870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.209821    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:08.209870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.209830    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.209972 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.209880    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:09.20986699 +0000 UTC m=+4.165914597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.438475 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:08.438371    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:12:07 +0000 UTC" deadline="2028-03-23 22:50:12.8779043 +0000 UTC"
Sep 11 16:17:08.438475 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:08.438414    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13422h33m4.439495546s"
Sep 11 16:17:08.471849 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:08.471817    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:08.472007 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:08.471957    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:08.578992 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.578968    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca53fdc1_656c_4a18_b255_78646e6fb4f9.slice/crio-5855ae2802589e3d36005f1d2d5353a543d9ee7b23d2443f84fbb83e19d4487b WatchSource:0}: Error finding container 5855ae2802589e3d36005f1d2d5353a543d9ee7b23d2443f84fbb83e19d4487b: Status 404 returned error can't find the container with id 5855ae2802589e3d36005f1d2d5353a543d9ee7b23d2443f84fbb83e19d4487b
Sep 11 16:17:08.580671 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.580650    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ee5227_cc0b_4691_a0be_4334a05612dc.slice/crio-4bac6cdd8af90faac8f2f3d8223c13ab3c2bac24619c177cd88b1163b89a675f WatchSource:0}: Error finding container 4bac6cdd8af90faac8f2f3d8223c13ab3c2bac24619c177cd88b1163b89a675f: Status 404 returned error can't find the container with id 4bac6cdd8af90faac8f2f3d8223c13ab3c2bac24619c177cd88b1163b89a675f
Sep 11 16:17:08.583977 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.583828    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3306db7f_f969_4cac_a413_5a9400e48e99.slice/crio-0731043e7aa88ab6f32f404e34374535e6fb7daab3a10eef2a16b744eeeb02dd WatchSource:0}: Error finding container 0731043e7aa88ab6f32f404e34374535e6fb7daab3a10eef2a16b744eeeb02dd: Status 404 returned error can't find the container with id 0731043e7aa88ab6f32f404e34374535e6fb7daab3a10eef2a16b744eeeb02dd
Sep 11 16:17:08.608463 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.608393    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ca5cb4_e2e6_4262_bfbc_d127bd6067ca.slice/crio-9694346d402af82b41a8c97ae7a787839d9fa39f748daaea089e6eb3e2d7882d WatchSource:0}: Error finding container 9694346d402af82b41a8c97ae7a787839d9fa39f748daaea089e6eb3e2d7882d: Status 404 returned error can't find the container with id 9694346d402af82b41a8c97ae7a787839d9fa39f748daaea089e6eb3e2d7882d
Sep 11 16:17:08.609864 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.609840    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9341296d_c6a8_495b_8413_1758fbc4d091.slice/crio-6dca2cf611380ae1d8385475a6608bbc225ebf0fd75c22fb607ca9f98bb99def WatchSource:0}: Error finding container 6dca2cf611380ae1d8385475a6608bbc225ebf0fd75c22fb607ca9f98bb99def: Status 404 returned error can't find the container with id 6dca2cf611380ae1d8385475a6608bbc225ebf0fd75c22fb607ca9f98bb99def
Sep 11 16:17:08.611308 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.611283    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ffbbfc_30e6_4dfe_815d_713685253804.slice/crio-61c084fc74a9559ade8b17234a8287e7bb159015a203c5fbce2c4483f321e7bc WatchSource:0}: Error finding container 61c084fc74a9559ade8b17234a8287e7bb159015a203c5fbce2c4483f321e7bc: Status 404 returned error can't find the container with id 61c084fc74a9559ade8b17234a8287e7bb159015a203c5fbce2c4483f321e7bc
Sep 11 16:17:08.612548 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.612529    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb902a3c9_3153_4972_a0d5_eff6e6aec791.slice/crio-7a445c3877cbd242fcfe8cbbcaa34c4f6393fa36dedb9ce38ee29e7df26e59d2 WatchSource:0}: Error finding container 7a445c3877cbd242fcfe8cbbcaa34c4f6393fa36dedb9ce38ee29e7df26e59d2: Status 404 returned error can't find the container with id 7a445c3877cbd242fcfe8cbbcaa34c4f6393fa36dedb9ce38ee29e7df26e59d2
Sep 11 16:17:08.613476 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.613445    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a4f2f1_7987_46fa_ae27_600b250b0d12.slice/crio-265dff2da425747e8434eb37d1885001b40f1221dcd11bf6c370c2462ed9db78 WatchSource:0}: Error finding container 265dff2da425747e8434eb37d1885001b40f1221dcd11bf6c370c2462ed9db78: Status 404 returned error can't find the container with id 265dff2da425747e8434eb37d1885001b40f1221dcd11bf6c370c2462ed9db78
Sep 11 16:17:08.614919 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:08.614674    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3425dd2f_36b9_4a54_bdcb_1989ae091941.slice/crio-963720208e39b45a72991bc7ad600ad7a94f7440ae687d341c0162700bfd948a WatchSource:0}: Error finding container 963720208e39b45a72991bc7ad600ad7a94f7440ae687d341c0162700bfd948a: Status 404 returned error can't find the container with id 963720208e39b45a72991bc7ad600ad7a94f7440ae687d341c0162700bfd948a
Sep 11 16:17:09.015539 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.015500    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:09.015769 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.015749    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:09.015849 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.015823    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:11.015802898 +0000 UTC m=+5.971850520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:09.116655 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.116497    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:09.116832 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.116807    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:09.116899 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.116886    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:11.116867313 +0000 UTC m=+6.072914939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:09.217792 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.217756    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:09.217985 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.217967    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:09.218057 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.217996    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:09.218057 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.218009    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:09.218158 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.218069    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:11.218048842 +0000 UTC m=+6.174096465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:09.439689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.438941    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:12:07 +0000 UTC" deadline="2028-03-11 02:03:49.505241169 +0000 UTC"
Sep 11 16:17:09.439689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.438984    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13113h46m40.066262637s"
Sep 11 16:17:09.477668 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.477641    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:09.477806 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.477775    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:09.478235 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.478213    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:09.478339 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:09.478308    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:09.489161 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.489103    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal" event={"ID":"520fabedc60a64e4fd6648fb59243517","Type":"ContainerStarted","Data":"d6fb9c99f37a781b6d39efcb0e70e784c99ca2ec1a3a5759e4c90cad69017d9d"}
Sep 11 16:17:09.506358 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.506326    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wbwf" event={"ID":"3425dd2f-36b9-4a54-bdcb-1989ae091941","Type":"ContainerStarted","Data":"963720208e39b45a72991bc7ad600ad7a94f7440ae687d341c0162700bfd948a"}
Sep 11 16:17:09.510946 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.510884    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jx4cp" event={"ID":"b902a3c9-3153-4972-a0d5-eff6e6aec791","Type":"ContainerStarted","Data":"7a445c3877cbd242fcfe8cbbcaa34c4f6393fa36dedb9ce38ee29e7df26e59d2"}
Sep 11 16:17:09.523514 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.523487    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vg278" event={"ID":"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca","Type":"ContainerStarted","Data":"9694346d402af82b41a8c97ae7a787839d9fa39f748daaea089e6eb3e2d7882d"}
Sep 11 16:17:09.526507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.526448    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwqsr" event={"ID":"d5ee5227-cc0b-4691-a0be-4334a05612dc","Type":"ContainerStarted","Data":"4bac6cdd8af90faac8f2f3d8223c13ab3c2bac24619c177cd88b1163b89a675f"}
Sep 11 16:17:09.531053 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.530984    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qk8cl" event={"ID":"3306db7f-f969-4cac-a413-5a9400e48e99","Type":"ContainerStarted","Data":"0731043e7aa88ab6f32f404e34374535e6fb7daab3a10eef2a16b744eeeb02dd"}
Sep 11 16:17:09.536589 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.535816    2550 generic.go:356] "Generic (PLEG): container finished" podID="4a048dfbc799f86a2baabe7fe5ea46cb" containerID="d2233f23f35e1d50f78ca32c33cf90d9839c4e60f2fc27d4cebff66cc4e7169f" exitCode=0
Sep 11 16:17:09.536589 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.535893    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal" event={"ID":"4a048dfbc799f86a2baabe7fe5ea46cb","Type":"ContainerDied","Data":"d2233f23f35e1d50f78ca32c33cf90d9839c4e60f2fc27d4cebff66cc4e7169f"}
Sep 11 16:17:09.544199 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.544153    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerStarted","Data":"265dff2da425747e8434eb37d1885001b40f1221dcd11bf6c370c2462ed9db78"}
Sep 11 16:17:09.555535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.555484    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-84.ec2.internal" podStartSLOduration=2.555467073 podStartE2EDuration="2.555467073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:09.504914516 +0000 UTC m=+4.460962150" watchObservedRunningTime="2026-09-11 16:17:09.555467073 +0000 UTC m=+4.511514705"
Sep 11 16:17:09.557286 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.557181    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"61c084fc74a9559ade8b17234a8287e7bb159015a203c5fbce2c4483f321e7bc"}
Sep 11 16:17:09.571566 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.571537    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d" event={"ID":"9341296d-c6a8-495b-8413-1758fbc4d091","Type":"ContainerStarted","Data":"6dca2cf611380ae1d8385475a6608bbc225ebf0fd75c22fb607ca9f98bb99def"}
Sep 11 16:17:09.580636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:09.580594    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xxs9n" event={"ID":"ca53fdc1-656c-4a18-b255-78646e6fb4f9","Type":"ContainerStarted","Data":"5855ae2802589e3d36005f1d2d5353a543d9ee7b23d2443f84fbb83e19d4487b"}
Sep 11 16:17:10.471901 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:10.471855    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:10.472533 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:10.472008    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:10.608479 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:10.608289    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal" event={"ID":"4a048dfbc799f86a2baabe7fe5ea46cb","Type":"ContainerStarted","Data":"5b39567f942404220002bb4e651bd31425d66efcd7387d92df350e01f21fcb24"}
Sep 11 16:17:10.624486 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:10.624429    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-84.ec2.internal" podStartSLOduration=3.624406632 podStartE2EDuration="3.624406632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:10.62366451 +0000 UTC m=+5.579712143" watchObservedRunningTime="2026-09-11 16:17:10.624406632 +0000 UTC m=+5.580454263"
Sep 11 16:17:11.035238 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:11.035196    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:11.035434 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.035399    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:11.035499 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.035472    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:15.035452249 +0000 UTC m=+9.991499880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:11.136453 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:11.136410    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:11.136689 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.136674    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:11.136783 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.136746    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:15.136723455 +0000 UTC m=+10.092771088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:11.237817 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:11.237772    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:11.237994 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.237948    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:11.237994 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.237972    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:11.237994 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.237987    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:11.238163 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.238050    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:15.238031609 +0000 UTC m=+10.194079233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:11.472997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:11.472146    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:11.472997 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.472287    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:11.472997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:11.472756    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:11.472997 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:11.472894    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:12.471419 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:12.471381    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:12.471640 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:12.471546    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:13.472725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:13.472693    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:13.473198 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:13.472843    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:13.473409 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:13.473389    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:13.473512 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:13.473491    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:14.472001 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:14.471915    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:14.472166 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:14.472026    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:15.069523 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:15.069491    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:15.069957 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.069647    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:15.069957 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.069714    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:23.069695759 +0000 UTC m=+18.025743382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:15.170324 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:15.170292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:15.170497 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.170420    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:15.170497 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.170477    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:23.170459385 +0000 UTC m=+18.126506994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:15.273573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:15.271820    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:15.273573 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.272781    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:15.273573 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.272813    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:15.273573 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.272831    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:15.273573 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.272922    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:23.272901858 +0000 UTC m=+18.228949466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:15.472839 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:15.472577    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:15.472996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:15.472639    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:15.472996 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.472877    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:15.472996 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:15.472975    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:16.471363 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:16.471308    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:16.471876 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:16.471492    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:17.471623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:17.471589    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:17.472045 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:17.471654    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:17.472045 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:17.471736    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:17.472176 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:17.472119    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:18.471376 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:18.471341    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:18.471574 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:18.471468    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:19.472092 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:19.472008    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:19.472092 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:19.472030    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:19.472576 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:19.472118    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:19.472576 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:19.472268    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:20.471529 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:20.471488    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:20.471698 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:20.471616    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:21.471841 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:21.471810    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:21.472292 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:21.471932    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:21.472292 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:21.472008    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:21.472292 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:21.472106    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:22.471468 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:22.471441    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:22.471695 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:22.471592    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:23.132389 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:23.132350    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:23.132826 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.132512    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:23.132826 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.132601    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:17:39.13257952 +0000 UTC m=+34.088627129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:23.233515 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:23.233482    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:23.233698 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.233659    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:23.233761 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.233734    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:39.233712995 +0000 UTC m=+34.189760618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:23.334142 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:23.334111    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:23.334312 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.334256    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:23.334312 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.334279    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:23.334312 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.334293    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:23.334450 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.334348    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:17:39.33433486 +0000 UTC m=+34.290382466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:23.471557 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:23.471444    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:23.471728 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.471585    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:23.471728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:23.471634    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:23.471835 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:23.471744    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:24.471998 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:24.471968    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:24.472428 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:24.472097    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:25.472427 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:25.472388    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:25.472864 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:25.472475    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:25.472864 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:25.472498    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:25.472864 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:25.472594    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:26.473125 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.471507    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:26.473125 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:26.471657    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:26.643031 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.642998    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wbwf" event={"ID":"3425dd2f-36b9-4a54-bdcb-1989ae091941","Type":"ContainerStarted","Data":"b19fe20788c7a3a5118106885076d57b89f4668eb62f51d2605cfd1a368f0e9e"}
Sep 11 16:17:26.644571 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.644531    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jx4cp" event={"ID":"b902a3c9-3153-4972-a0d5-eff6e6aec791","Type":"ContainerStarted","Data":"e8a614cca06e19858fb585edfe890958ca47a8062a5d79e8feec4e5701d77461"}
Sep 11 16:17:26.646056 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.646036    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vg278" event={"ID":"c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca","Type":"ContainerStarted","Data":"28cb004014965100d5223d485778e8395db4d1aad24612d66c6a79f246aa1990"}
Sep 11 16:17:26.647695 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.647673    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lwqsr" event={"ID":"d5ee5227-cc0b-4691-a0be-4334a05612dc","Type":"ContainerStarted","Data":"49b5f013a96704a9a53d068b47de3299231d7ecfa25d99426db05d2f352024ea"}
Sep 11 16:17:26.649148 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.649118    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerStarted","Data":"23c46ba6db2ce3e287c628b5ea7814a9dc4482e63ca118f323b2e045b3ed41ca"}
Sep 11 16:17:26.652193 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.652172    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"cd4a2aa12b233e63144662f72a211b94c0e9528a9c1b8ce7f538a9c57c26e8cc"}
Sep 11 16:17:26.653728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.653705    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d" event={"ID":"9341296d-c6a8-495b-8413-1758fbc4d091","Type":"ContainerStarted","Data":"43eb0122e1b7a7eb4c4687891a1d3cce0fe7001b5fae0f60e1ab4bf97434a592"}
Sep 11 16:17:26.662810 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.662770    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wbwf" podStartSLOduration=2.956214598 podStartE2EDuration="20.662758701s" totalImagesPullingTime="17.706544103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.662014739 +0000 UTC m=+21.618062368" watchObservedRunningTime="2026-09-11 16:17:26.662758701 +0000 UTC m=+21.618806329"
Sep 11 16:17:26.681487 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.681451    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lwqsr" podStartSLOduration=2.920777994 podStartE2EDuration="20.681440152s" totalImagesPullingTime="17.760662158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.681351986 +0000 UTC m=+21.637399608" watchObservedRunningTime="2026-09-11 16:17:26.681440152 +0000 UTC m=+21.637487780"
Sep 11 16:17:26.721174 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.721134    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vg278" podStartSLOduration=3.274986161 podStartE2EDuration="20.721120368s" totalImagesPullingTime="17.446134207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.720521126 +0000 UTC m=+21.676568755" watchObservedRunningTime="2026-09-11 16:17:26.721120368 +0000 UTC m=+21.677168057"
Sep 11 16:17:26.721289 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:26.721271    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jx4cp" podStartSLOduration=2.993909227 podStartE2EDuration="20.72126604s" totalImagesPullingTime="17.727356813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:26.698382643 +0000 UTC m=+21.654430272" watchObservedRunningTime="2026-09-11 16:17:26.72126604 +0000 UTC m=+21.677313729"
Sep 11 16:17:27.472335 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.472154    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:27.472504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.472205    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:27.472504 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:27.472373    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:27.472504 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:27.472438    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:27.656654 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.656621    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qk8cl" event={"ID":"3306db7f-f969-4cac-a413-5a9400e48e99","Type":"ContainerStarted","Data":"05d7e9ee1514d7c7d0749585a156d6acf350ff472a9274426cf2ab3212b5d721"}
Sep 11 16:17:27.658109 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.658088    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1a4f2f1-7987-46fa-ae27-600b250b0d12" containerID="23c46ba6db2ce3e287c628b5ea7814a9dc4482e63ca118f323b2e045b3ed41ca" exitCode=0
Sep 11 16:17:27.658222 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.658153    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerDied","Data":"23c46ba6db2ce3e287c628b5ea7814a9dc4482e63ca118f323b2e045b3ed41ca"}
Sep 11 16:17:27.660802 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.660787    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:17:27.661126 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.661109    2550 generic.go:356] "Generic (PLEG): container finished" podID="92ffbbfc-30e6-4dfe-815d-713685253804" containerID="93480a57346306e87b45c1afe6e6d939ea84bc16d9497085cf2c342366126b72" exitCode=1
Sep 11 16:17:27.661217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.661195    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"008e28dbf12b5e95af8624379b72fff4887de84570049a953c829f61544217c1"}
Sep 11 16:17:27.661271 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.661260    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"07be11027772af0d39d6c7ee7db8c4f216cec97e3baf3c4a6e15494e4a7f5e77"}
Sep 11 16:17:27.661306 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.661296    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"05ca6be39b6037cfa8cf247d6bb5234dfa31ed9958ec2fc23cf4f19a14f39dde"}
Sep 11 16:17:27.661340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.661333    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"bad253c15eeaa0290640f79cf82919dd441bff627b9fcdc727d844ca2b52c1ac"}
Sep 11 16:17:27.661392 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.661361    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerDied","Data":"93480a57346306e87b45c1afe6e6d939ea84bc16d9497085cf2c342366126b72"}
Sep 11 16:17:27.681268 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.681216    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qk8cl" podStartSLOduration=3.61812571 podStartE2EDuration="21.681204473s" totalImagesPullingTime="18.063078763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:27.675413657 +0000 UTC m=+22.631461284" watchObservedRunningTime="2026-09-11 16:17:27.681204473 +0000 UTC m=+22.637252100"
Sep 11 16:17:27.794928 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:27.794800    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:17:28.227070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.226922    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:28.227655 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.227633    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:28.448576 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.448454    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:17:27.794925282Z","UUID":"6dbb3405-3aec-40bb-838f-7aafffb85e13","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:17:28.450098 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.450076    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:17:28.450212 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.450107    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:17:28.471262 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.471237    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:28.471397 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:28.471369    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:28.665712 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.665641    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d" event={"ID":"9341296d-c6a8-495b-8413-1758fbc4d091","Type":"ContainerStarted","Data":"711ab8c5752b56f18df951f336c91d82cd6630f0a21e29d64a1f2ef00d7490cd"}
Sep 11 16:17:28.667344 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:28.667309    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xxs9n" event={"ID":"ca53fdc1-656c-4a18-b255-78646e6fb4f9","Type":"ContainerStarted","Data":"569c864216776e5db72f49b43c2196ad6d1d86ddc00ad6a9d9927728b9133ccb"}
Sep 11 16:17:29.471900 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.471821    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:29.472126 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:29.471952    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:29.472126 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.472010    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:29.472208 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:29.472189    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:29.673710 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.673675    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:17:29.674131 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.674103    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"517a2794bb911f366f3b83c17138abd619528d768571d5fab06d3a85831dac43"}
Sep 11 16:17:29.676463 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.676439    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d" event={"ID":"9341296d-c6a8-495b-8413-1758fbc4d091","Type":"ContainerStarted","Data":"1099fb7fdf79574a7235e57c9510933f8f326c5c3a0e162a10e1eed679686d37"}
Sep 11 16:17:29.676616 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.676496    2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 16:17:29.695871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.695833    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bvc7d" podStartSLOduration=3.927306148 podStartE2EDuration="23.695821073s" totalImagesPullingTime="19.768514925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:29.695790198 +0000 UTC m=+24.651837826" watchObservedRunningTime="2026-09-11 16:17:29.695821073 +0000 UTC m=+24.651868702"
Sep 11 16:17:29.696179 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:29.696154    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xxs9n" podStartSLOduration=5.930951787 podStartE2EDuration="23.696146748s" totalImagesPullingTime="17.765194961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:28.683659964 +0000 UTC m=+23.639707591" watchObservedRunningTime="2026-09-11 16:17:29.696146748 +0000 UTC m=+24.652194375"
Sep 11 16:17:30.471678 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:30.471647    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:30.471844 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:30.471769    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:31.472073 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:31.471989    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:31.472073 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:31.472014    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:31.472571 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:31.472116    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:31.472571 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:31.472229    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:32.471689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.471484    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:32.471807 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:32.471724    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:32.683960 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.683928    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1a4f2f1-7987-46fa-ae27-600b250b0d12" containerID="7cfdaf8465fbe7e4c0bbb25cef98e4e8a6557c542a8d7559a9a58b4583293bbb" exitCode=0
Sep 11 16:17:32.684566 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.684010    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerDied","Data":"7cfdaf8465fbe7e4c0bbb25cef98e4e8a6557c542a8d7559a9a58b4583293bbb"}
Sep 11 16:17:32.687669 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.687631    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:17:32.688042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.688020    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"6320cdd30426657a7cf53305dc07a09b9e68df830aa90e3cbfbf11ca2e5f6e6d"}
Sep 11 16:17:32.688233 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.688217    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:32.688304 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.688292    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:32.688401 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.688346    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:32.688446 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.688428    2550 scope.go:117] "RemoveContainer" containerID="93480a57346306e87b45c1afe6e6d939ea84bc16d9497085cf2c342366126b72"
Sep 11 16:17:32.703379 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.703351    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:32.704178 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:32.704156    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:17:33.358875 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.358849    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:33.359086 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.358970    2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 16:17:33.359700 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.359593    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vg278"
Sep 11 16:17:33.471620 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.471413    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:33.471724 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.471453    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:33.471789 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:33.471729    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:33.471789 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:33.471751    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:33.599238 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.599213    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-754qk"]
Sep 11 16:17:33.602465 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.602444    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vcxfz"]
Sep 11 16:17:33.602609 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.602567    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:33.602740 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:33.602716    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:33.603280 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.603244    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bxrwt"]
Sep 11 16:17:33.691973 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.691882    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1a4f2f1-7987-46fa-ae27-600b250b0d12" containerID="7fa8e10b4c77f58c3f0631116048662db66c5fe647593683d11d0d83713ca2c1" exitCode=0
Sep 11 16:17:33.691973 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.691965    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerDied","Data":"7fa8e10b4c77f58c3f0631116048662db66c5fe647593683d11d0d83713ca2c1"}
Sep 11 16:17:33.695989 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.695971    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:17:33.696373 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.696352    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:33.696463 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.696372    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" event={"ID":"92ffbbfc-30e6-4dfe-815d-713685253804","Type":"ContainerStarted","Data":"6da16ba7b8a517ca294262ccd3f9a637238c4ccbf6239bd75a2f454ffe676748"}
Sep 11 16:17:33.696463 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:33.696432    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:33.696656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.696645    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:33.696708 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:33.696696    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:33.740979 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:33.740940    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt" podStartSLOduration=10.008656778 podStartE2EDuration="27.740927888s" totalImagesPullingTime="17.73227111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:33.740399896 +0000 UTC m=+28.696447526" watchObservedRunningTime="2026-09-11 16:17:33.740927888 +0000 UTC m=+28.696975509"
Sep 11 16:17:35.471880 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:35.471844    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:35.472296 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:35.471923    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:35.472296 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:35.471940    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:35.472296 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:35.472026    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:35.472296 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:35.472027    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:35.472296 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:35.472120    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:35.703189 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:35.703104    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1a4f2f1-7987-46fa-ae27-600b250b0d12" containerID="b0502567693df24c0f8b569199458552d210b3f21e2582be1ff4629d64aec21b" exitCode=0
Sep 11 16:17:35.703311 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:35.703186    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerDied","Data":"b0502567693df24c0f8b569199458552d210b3f21e2582be1ff4629d64aec21b"}
Sep 11 16:17:37.471537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:37.471504    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:37.471995 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:37.471513    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:37.471995 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:37.471639    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-754qk" podUID="231ac950-39f0-4afb-a966-3414a5124d8b"
Sep 11 16:17:37.471995 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:37.471661    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:37.471995 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:37.471745    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bxrwt" podUID="60b9574a-1c21-4f49-a471-4848b586bf0d"
Sep 11 16:17:37.471995 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:37.471847    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcxfz" podUID="dac1c7b7-7288-434a-8196-cd1a17d2f0b8"
Sep 11 16:17:38.311757 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.311686    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-84.ec2.internal" event="NodeReady"
Sep 11 16:17:38.311899 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.311803    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:17:38.365316 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.365290    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pgtfg"]
Sep 11 16:17:38.390144 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.390120    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vdntq"]
Sep 11 16:17:38.390289 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.390242    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.392759 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.392733    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:17:38.393188 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.393172    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-244pg\""
Sep 11 16:17:38.394075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.394060    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:17:38.408176 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.408155    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgtfg"]
Sep 11 16:17:38.408277 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.408201    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdntq"]
Sep 11 16:17:38.408344 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.408285    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.411354 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.410511    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:17:38.411354 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.410991    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qbh4n\""
Sep 11 16:17:38.411354 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.411250    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:17:38.411573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.411538    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:17:38.554262 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.554225    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61c0e3ee-7239-48a6-8823-82be2e360cae-config-volume\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.554262 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.554258    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.554728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.554288    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspcf\" (UniqueName: \"kubernetes.io/projected/61c0e3ee-7239-48a6-8823-82be2e360cae-kube-api-access-mspcf\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.554728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.554384    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c0e3ee-7239-48a6-8823-82be2e360cae-tmp-dir\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.554728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.554430    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wklcg\" (UniqueName: \"kubernetes.io/projected/379fc8b9-b464-4ce9-bf17-882739302bdf-kube-api-access-wklcg\") pod \"ingress-canary-vdntq\" (UID: \"379fc8b9-b464-4ce9-bf17-882739302bdf\") " pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.554728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.554478    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379fc8b9-b464-4ce9-bf17-882739302bdf-cert\") pod \"ingress-canary-vdntq\" (UID: \"379fc8b9-b464-4ce9-bf17-882739302bdf\") " pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.655613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.655517    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61c0e3ee-7239-48a6-8823-82be2e360cae-config-volume\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.655613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.655575    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.655837 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.655622    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspcf\" (UniqueName: \"kubernetes.io/projected/61c0e3ee-7239-48a6-8823-82be2e360cae-kube-api-access-mspcf\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.655837 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.655654    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c0e3ee-7239-48a6-8823-82be2e360cae-tmp-dir\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.655837 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.655680    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wklcg\" (UniqueName: \"kubernetes.io/projected/379fc8b9-b464-4ce9-bf17-882739302bdf-kube-api-access-wklcg\") pod \"ingress-canary-vdntq\" (UID: \"379fc8b9-b464-4ce9-bf17-882739302bdf\") " pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.655837 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:38.655696    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:38.655837 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.655708    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379fc8b9-b464-4ce9-bf17-882739302bdf-cert\") pod \"ingress-canary-vdntq\" (UID: \"379fc8b9-b464-4ce9-bf17-882739302bdf\") " pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.655837 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:38.655767    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:39.155744877 +0000 UTC m=+34.111792487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:38.656181 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.656152    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/61c0e3ee-7239-48a6-8823-82be2e360cae-config-volume\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.656304 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.656271    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c0e3ee-7239-48a6-8823-82be2e360cae-tmp-dir\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.660211 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.660190    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379fc8b9-b464-4ce9-bf17-882739302bdf-cert\") pod \"ingress-canary-vdntq\" (UID: \"379fc8b9-b464-4ce9-bf17-882739302bdf\") " pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.665193 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.665171    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspcf\" (UniqueName: \"kubernetes.io/projected/61c0e3ee-7239-48a6-8823-82be2e360cae-kube-api-access-mspcf\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:38.665579 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.665561    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wklcg\" (UniqueName: \"kubernetes.io/projected/379fc8b9-b464-4ce9-bf17-882739302bdf-kube-api-access-wklcg\") pod \"ingress-canary-vdntq\" (UID: \"379fc8b9-b464-4ce9-bf17-882739302bdf\") " pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:38.721293 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:38.721268    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdntq"
Sep 11 16:17:39.160389 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.160351    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:39.160703 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.160443    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:39.160703 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.160464    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:39.160703 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.160552    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.160521636 +0000 UTC m=+35.116569242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:39.160703 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.160555    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:39.160703 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.160598    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret podName:231ac950-39f0-4afb-a966-3414a5124d8b nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.160586844 +0000 UTC m=+66.116634450 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret") pod "global-pull-secret-syncer-754qk" (UID: "231ac950-39f0-4afb-a966-3414a5124d8b") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:39.261665 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.261630    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:39.261930 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.261800    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:39.261930 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.261883    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.261861973 +0000 UTC m=+66.217909580 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:39.362454 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.362422    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:39.362692 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.362632    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:39.362692 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.362678    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:39.362692 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.362689    2550 projected.go:196] Error preparing data for projected volume kube-api-access-5js57 for pod openshift-network-diagnostics/network-check-target-bxrwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:39.362869 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:39.362740    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57 podName:60b9574a-1c21-4f49-a471-4848b586bf0d nodeName:}" failed. No retries permitted until 2026-09-11 16:18:11.362725705 +0000 UTC m=+66.318773311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5js57" (UniqueName: "kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57") pod "network-check-target-bxrwt" (UID: "60b9574a-1c21-4f49-a471-4848b586bf0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:39.471677 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.471607    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:17:39.471872 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.471607    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:17:39.471872 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.471784    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:17:39.475293 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.475273    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:39.475428 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.475274    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:39.475758 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.475740    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks2m\""
Sep 11 16:17:39.475758 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.475748    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:39.475758 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.475757    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrmtk\""
Sep 11 16:17:39.475983 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.475965    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:39.944362 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:39.944203    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdntq"]
Sep 11 16:17:39.947447 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:17:39.947422    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379fc8b9_b464_4ce9_bf17_882739302bdf.slice/crio-8455337567a2b36ea7f4257816964bc809da0490c9ae2c82545ae5ff367e5226 WatchSource:0}: Error finding container 8455337567a2b36ea7f4257816964bc809da0490c9ae2c82545ae5ff367e5226: Status 404 returned error can't find the container with id 8455337567a2b36ea7f4257816964bc809da0490c9ae2c82545ae5ff367e5226
Sep 11 16:17:40.168490 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:40.168462    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:40.168683 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:40.168660    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:40.168774 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:40.168759    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:42.16873523 +0000 UTC m=+37.124782903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:40.716484 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:40.716445    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdntq" event={"ID":"379fc8b9-b464-4ce9-bf17-882739302bdf","Type":"ContainerStarted","Data":"8455337567a2b36ea7f4257816964bc809da0490c9ae2c82545ae5ff367e5226"}
Sep 11 16:17:40.719508 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:40.719472    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1a4f2f1-7987-46fa-ae27-600b250b0d12" containerID="1c6263b7ec4a9b931f6c16419c26ab2b81b2a641e99b42a9b42815a12bdd69b8" exitCode=0
Sep 11 16:17:40.719669 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:40.719513    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerDied","Data":"1c6263b7ec4a9b931f6c16419c26ab2b81b2a641e99b42a9b42815a12bdd69b8"}
Sep 11 16:17:41.726579 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:41.726518    2550 generic.go:356] "Generic (PLEG): container finished" podID="b1a4f2f1-7987-46fa-ae27-600b250b0d12" containerID="edd46b509603ca83deaaa2064363298a4c2efc4631ad9a42527d531e8a9aca4c" exitCode=0
Sep 11 16:17:41.727338 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:41.726597    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerDied","Data":"edd46b509603ca83deaaa2064363298a4c2efc4631ad9a42527d531e8a9aca4c"}
Sep 11 16:17:42.183822 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:42.183777    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:42.183979 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:42.183916    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:42.183979 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:42.183978    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:46.183961576 +0000 UTC m=+41.140009195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:42.732829 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:42.732695    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlccg" event={"ID":"b1a4f2f1-7987-46fa-ae27-600b250b0d12","Type":"ContainerStarted","Data":"0ac78ef3d641778c638b0bc2af28bbe12409b248b19cca0c30919184dc67bdf0"}
Sep 11 16:17:42.734162 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:42.734134    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdntq" event={"ID":"379fc8b9-b464-4ce9-bf17-882739302bdf","Type":"ContainerStarted","Data":"ef67665b741100fcc4679cc08a62c7250e6875faba4d29868401fce51a4ecf2f"}
Sep 11 16:17:42.756660 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:42.756593    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dlccg" podStartSLOduration=8.831676847 podStartE2EDuration="36.756559881s" totalImagesPullingTime="27.924883034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:42.755253671 +0000 UTC m=+37.711301300" watchObservedRunningTime="2026-09-11 16:17:42.756559881 +0000 UTC m=+37.712607499"
Sep 11 16:17:42.769713 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:42.769675    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vdntq" podStartSLOduration=2.680870735 podStartE2EDuration="4.769662519s" totalImagesPullingTime="2.088791784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:42.768901566 +0000 UTC m=+37.724949194" watchObservedRunningTime="2026-09-11 16:17:42.769662519 +0000 UTC m=+37.725710147"
Sep 11 16:17:46.212063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:46.212024    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:46.212470 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:46.212146    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:46.212470 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:46.212212    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:17:54.212194843 +0000 UTC m=+49.168242467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:54.263200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:17:54.263165    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:17:54.263645 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:54.263308    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:54.263645 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:17:54.263374    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:18:10.263359602 +0000 UTC m=+65.219407208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:18:04.717774 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:04.717743    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s6jwt"
Sep 11 16:18:10.266037 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:10.265997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:18:10.270314 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:10.266138    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:18:10.270314 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:10.266207    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:18:42.266191002 +0000 UTC m=+97.222238621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:18:11.170943 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.170906    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:18:11.173302 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.173285    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:18:11.185184 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.185160    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/231ac950-39f0-4afb-a966-3414a5124d8b-original-pull-secret\") pod \"global-pull-secret-syncer-754qk\" (UID: \"231ac950-39f0-4afb-a966-3414a5124d8b\") " pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:18:11.271949 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.271924    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:18:11.274344 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.274329    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:18:11.282247 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:11.282230    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 16:18:11.282793 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:11.282770    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs podName:dac1c7b7-7288-434a-8196-cd1a17d2f0b8 nodeName:}" failed. No retries permitted until 2026-09-11 16:19:15.282741835 +0000 UTC m=+130.238789457 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs") pod "network-metrics-daemon-vcxfz" (UID: "dac1c7b7-7288-434a-8196-cd1a17d2f0b8") : secret "metrics-daemon-secret" not found
Sep 11 16:18:11.285668 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.285646    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-754qk"
Sep 11 16:18:11.373010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.372979    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:18:11.374972 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.374950    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:18:11.385370 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.385351    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:18:11.396071 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.396045    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5js57\" (UniqueName: \"kubernetes.io/projected/60b9574a-1c21-4f49-a471-4848b586bf0d-kube-api-access-5js57\") pod \"network-check-target-bxrwt\" (UID: \"60b9574a-1c21-4f49-a471-4848b586bf0d\") " pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:18:11.585263 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.585238    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-754qk"]
Sep 11 16:18:11.589199 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:11.589176    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231ac950_39f0_4afb_a966_3414a5124d8b.slice/crio-882d94454a15bf63fff3c8de342c0e350db87d20d76fcea782463ec39cc8aa06 WatchSource:0}: Error finding container 882d94454a15bf63fff3c8de342c0e350db87d20d76fcea782463ec39cc8aa06: Status 404 returned error can't find the container with id 882d94454a15bf63fff3c8de342c0e350db87d20d76fcea782463ec39cc8aa06
Sep 11 16:18:11.592189 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.592167    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrmtk\""
Sep 11 16:18:11.600865 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.600848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:18:11.798148 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.798117    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-754qk" event={"ID":"231ac950-39f0-4afb-a966-3414a5124d8b","Type":"ContainerStarted","Data":"882d94454a15bf63fff3c8de342c0e350db87d20d76fcea782463ec39cc8aa06"}
Sep 11 16:18:11.894413 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:11.894349    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bxrwt"]
Sep 11 16:18:11.897228 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:11.897194    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b9574a_1c21_4f49_a471_4848b586bf0d.slice/crio-ec4d999d5747fcaabe005c1c19d5a46243fa645436500d1eb8d2a25b0556701c WatchSource:0}: Error finding container ec4d999d5747fcaabe005c1c19d5a46243fa645436500d1eb8d2a25b0556701c: Status 404 returned error can't find the container with id ec4d999d5747fcaabe005c1c19d5a46243fa645436500d1eb8d2a25b0556701c
Sep 11 16:18:12.803507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:12.803429    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bxrwt" event={"ID":"60b9574a-1c21-4f49-a471-4848b586bf0d","Type":"ContainerStarted","Data":"ec4d999d5747fcaabe005c1c19d5a46243fa645436500d1eb8d2a25b0556701c"}
Sep 11 16:18:16.815961 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:16.815920    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-754qk" event={"ID":"231ac950-39f0-4afb-a966-3414a5124d8b","Type":"ContainerStarted","Data":"62ee92336ad319611257d99cb9cdb4b2b5446294f9f791e5b4d4abf0e33643fa"}
Sep 11 16:18:16.817334 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:16.817310    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bxrwt" event={"ID":"60b9574a-1c21-4f49-a471-4848b586bf0d","Type":"ContainerStarted","Data":"72f74e5f21a2c7660429f3a7720508429c85b8c730287dfd68ec9b9bbc15ef7f"}
Sep 11 16:18:16.817466 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:16.817367    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:18:16.832692 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:16.832650    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-754qk" podStartSLOduration=66.379444032 podStartE2EDuration="1m10.832636445s" totalImagesPullingTime="4.453192413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:16.830214823 +0000 UTC m=+71.786262463" watchObservedRunningTime="2026-09-11 16:18:16.832636445 +0000 UTC m=+71.788684072"
Sep 11 16:18:16.846402 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:16.846364    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bxrwt" podStartSLOduration=66.704323955 podStartE2EDuration="1m10.846351585s" totalImagesPullingTime="4.14202763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:16.845933623 +0000 UTC m=+71.801981251" watchObservedRunningTime="2026-09-11 16:18:16.846351585 +0000 UTC m=+71.802399212"
Sep 11 16:18:19.475157 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.475104    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"]
Sep 11 16:18:19.476904 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.476889    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.478937 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.478914    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 16:18:19.479046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.478961    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:18:19.481091 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.481070    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:18:19.481091 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.481084    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 16:18:19.481226 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.481074    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x9d4n\""
Sep 11 16:18:19.497870 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.497850    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"]
Sep 11 16:18:19.522217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.522192    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.522313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.522241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/076fb9d4-54bd-41e5-9c42-8a9fb362020f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.522313 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.522297    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5pg\" (UniqueName: \"kubernetes.io/projected/076fb9d4-54bd-41e5-9c42-8a9fb362020f-kube-api-access-ng5pg\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.587325 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.587303    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78b74d9c58-fgt69"]
Sep 11 16:18:19.588934 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.588920    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.594684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.594661    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 16:18:19.594786 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.594711    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 16:18:19.594786 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.594751    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j4zhm\""
Sep 11 16:18:19.595077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.595061    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 16:18:19.595185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.595120    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 16:18:19.595185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.595129    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 16:18:19.595185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.595164    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 16:18:19.603062 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.603036    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78b74d9c58-fgt69"]
Sep 11 16:18:19.623495 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623472    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.623591 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623499    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkjg\" (UniqueName: \"kubernetes.io/projected/ba8f0c3a-d449-423a-94ff-5b2ca617c983-kube-api-access-jgkjg\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.623591 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623518    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.623591 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623537    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-stats-auth\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.623693 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/076fb9d4-54bd-41e5-9c42-8a9fb362020f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.623693 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:19.623654    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:19.623754 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623693    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-default-certificate\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.623754 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:19.623711    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls podName:076fb9d4-54bd-41e5-9c42-8a9fb362020f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:20.123692142 +0000 UTC m=+75.079739762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ghwt8" (UID: "076fb9d4-54bd-41e5-9c42-8a9fb362020f") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:19.623824 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623789    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5pg\" (UniqueName: \"kubernetes.io/projected/076fb9d4-54bd-41e5-9c42-8a9fb362020f-kube-api-access-ng5pg\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.623856 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.623823    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.624368 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.624349    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/076fb9d4-54bd-41e5-9c42-8a9fb362020f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.636734 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.636711    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5pg\" (UniqueName: \"kubernetes.io/projected/076fb9d4-54bd-41e5-9c42-8a9fb362020f-kube-api-access-ng5pg\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:19.724852 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.724819    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-default-certificate\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.724943 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.724866    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.724990 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:19.724954    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:20.224941818 +0000 UTC m=+75.180989436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : configmap references non-existent config key: service-ca.crt
Sep 11 16:18:19.725052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.724982    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkjg\" (UniqueName: \"kubernetes.io/projected/ba8f0c3a-d449-423a-94ff-5b2ca617c983-kube-api-access-jgkjg\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.725052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.725022    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.725147 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.725056    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-stats-auth\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.725147 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:19.725134    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 16:18:19.725297 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:19.725171    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:20.225157074 +0000 UTC m=+75.181204695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : secret "router-metrics-certs-default" not found
Sep 11 16:18:19.727184 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.727162    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-default-certificate\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.727255 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.727203    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-stats-auth\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:19.733453 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:19.733433    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkjg\" (UniqueName: \"kubernetes.io/projected/ba8f0c3a-d449-423a-94ff-5b2ca617c983-kube-api-access-jgkjg\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:20.127243 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:20.127208    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:20.127407 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:20.127316    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:20.127407 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:20.127379    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls podName:076fb9d4-54bd-41e5-9c42-8a9fb362020f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:21.127364088 +0000 UTC m=+76.083411694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ghwt8" (UID: "076fb9d4-54bd-41e5-9c42-8a9fb362020f") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:20.228454 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:20.228418    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:20.228590 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:20.228467    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:20.228590 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:20.228584    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 16:18:20.228668 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:20.228631    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:21.228605557 +0000 UTC m=+76.184653177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : configmap references non-existent config key: service-ca.crt
Sep 11 16:18:20.228717 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:20.228670    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:21.228658676 +0000 UTC m=+76.184706297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : secret "router-metrics-certs-default" not found
Sep 11 16:18:21.135070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:21.135034    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:21.135518 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:21.135183    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:21.135518 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:21.135245    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls podName:076fb9d4-54bd-41e5-9c42-8a9fb362020f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:23.135231379 +0000 UTC m=+78.091278984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ghwt8" (UID: "076fb9d4-54bd-41e5-9c42-8a9fb362020f") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:21.236267 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:21.236229    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:21.236387 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:21.236280    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:21.236387 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:21.236355    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 16:18:21.236460 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:21.236383    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:23.236365585 +0000 UTC m=+78.192413191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : configmap references non-existent config key: service-ca.crt
Sep 11 16:18:21.236460 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:21.236442    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:23.236432188 +0000 UTC m=+78.192479794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : secret "router-metrics-certs-default" not found
Sep 11 16:18:23.150281 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.150250    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:23.150659 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.150349    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:23.150659 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.150401    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls podName:076fb9d4-54bd-41e5-9c42-8a9fb362020f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.150386503 +0000 UTC m=+82.106434108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ghwt8" (UID: "076fb9d4-54bd-41e5-9c42-8a9fb362020f") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:23.250976 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.250947    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:23.251076 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.250993    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:23.251113 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.251077    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 16:18:23.251113 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.251101    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.251085882 +0000 UTC m=+82.207133517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : configmap references non-existent config key: service-ca.crt
Sep 11 16:18:23.251187 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.251149    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.251141923 +0000 UTC m=+82.207189530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : secret "router-metrics-certs-default" not found
Sep 11 16:18:23.549609 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.549530    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9q95b"]
Sep 11 16:18:23.552031 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.552017    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.554063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.554034    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:18:23.554301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.554286    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:18:23.554359 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.554331    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:18:23.554359 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.554345    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26bj7\""
Sep 11 16:18:23.554437 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.554358    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:18:23.570725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.570706    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9q95b"]
Sep 11 16:18:23.653773 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.653752    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-data-volume\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.653866 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.653780    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.653866 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.653805    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-crio-socket\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.653943 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.653865    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6jh\" (UniqueName: \"kubernetes.io/projected/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-api-access-cj6jh\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.653943 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.653896    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.754712 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.754685    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6jh\" (UniqueName: \"kubernetes.io/projected/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-api-access-cj6jh\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.754811 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.754720    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.754811 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.754768    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-data-volume\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.754811 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.754784    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.754811 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.754805    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-crio-socket\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.755008 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.754883    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-crio-socket\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.755008 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.754896    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:23.755008 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.754926    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:23.755008 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.754963    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.254944664 +0000 UTC m=+79.210992293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:23.755008 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:23.754982    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.254972065 +0000 UTC m=+79.211019675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:23.755172 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.755137    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-data-volume\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:23.763759 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:23.763742    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6jh\" (UniqueName: \"kubernetes.io/projected/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-api-access-cj6jh\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:24.258729 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:24.258700    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:24.259087 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:24.258761    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:24.259087 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:24.258843    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:24.259087 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:24.258862    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:24.259087 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:24.258899    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:25.258885258 +0000 UTC m=+80.214932863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:24.259087 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:24.258918    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:25.258904186 +0000 UTC m=+80.214951792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.267472 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:25.267429    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:25.267870 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:25.267519    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:25.267870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:25.267570    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:25.267870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:25.267643    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.267628434 +0000 UTC m=+82.223676043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:25.267870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:25.267664    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.267870 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:25.267746    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.267732066 +0000 UTC m=+82.223779671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.660201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:25.660177    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jx4cp_b902a3c9-3153-4972-a0d5-eff6e6aec791/dns-node-resolver/0.log"
Sep 11 16:18:26.061452 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:26.061378    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wbwf_3425dd2f-36b9-4a54-bdcb-1989ae091941/node-ca/0.log"
Sep 11 16:18:27.180237 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:27.180203    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:27.180735 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.180370    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:27.180735 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.180449    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls podName:076fb9d4-54bd-41e5-9c42-8a9fb362020f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:35.180429412 +0000 UTC m=+90.136477038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ghwt8" (UID: "076fb9d4-54bd-41e5-9c42-8a9fb362020f") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:27.265673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:27.265651    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdntq_379fc8b9-b464-4ce9-bf17-882739302bdf/serve-healthcheck-canary/0.log"
Sep 11 16:18:27.280790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:27.280765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:27.280889 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:27.280798    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:27.280889 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:27.280833    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:27.280889 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:27.280856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:27.280889 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.280878    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:27.281021 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.280934    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.280918004 +0000 UTC m=+86.236965628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:27.281021 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.280946    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:27.281021 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.280952    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:35.280942762 +0000 UTC m=+90.236990369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : configmap references non-existent config key: service-ca.crt
Sep 11 16:18:27.281021 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.280986    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.280971239 +0000 UTC m=+86.237018859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:27.281021 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.281000    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 16:18:27.281185 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:27.281055    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:35.281041592 +0000 UTC m=+90.237089210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : secret "router-metrics-certs-default" not found
Sep 11 16:18:31.310466 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:31.310325    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:31.310466 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:31.310396    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:31.310912 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:31.310470    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:31.310912 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:31.310531    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.310516922 +0000 UTC m=+94.266564529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:31.310912 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:31.310470    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:31.310912 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:31.310645    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.310626137 +0000 UTC m=+94.266673765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:34.723705 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.723678    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"]
Sep 11 16:18:34.726180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.726165    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"
Sep 11 16:18:34.729706 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.729678    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 16:18:34.729706 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.729678    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:34.730052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.730039    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8gmtd\""
Sep 11 16:18:34.739689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.739670    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"]
Sep 11 16:18:34.826467 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.826439    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"]
Sep 11 16:18:34.828263 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.828248    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.833091 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.833054    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6s2\" (UniqueName: \"kubernetes.io/projected/7070d490-016f-478a-a093-778cc0eb45bd-kube-api-access-fb6s2\") pod \"migrator-8b4d697c5-p65wk\" (UID: \"7070d490-016f-478a-a093-778cc0eb45bd\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"
Sep 11 16:18:34.836283 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.836267    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 16:18:34.837209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.837192    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 16:18:34.837295 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.837277    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 16:18:34.837435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.837421    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 16:18:34.837473 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.837457    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 16:18:34.838030 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.838016    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 16:18:34.839551 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.839529    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 16:18:34.850746 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.850723    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"]
Sep 11 16:18:34.933759 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.933727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.933759 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.933761    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbqbz\" (UniqueName: \"kubernetes.io/projected/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-kube-api-access-vbqbz\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.933924 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.933788    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-hub\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.933924 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.933861    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.933924 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.933920    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6s2\" (UniqueName: \"kubernetes.io/projected/7070d490-016f-478a-a093-778cc0eb45bd-kube-api-access-fb6s2\") pod \"migrator-8b4d697c5-p65wk\" (UID: \"7070d490-016f-478a-a093-778cc0eb45bd\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"
Sep 11 16:18:34.934020 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.933974    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-ca\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.934020 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.934008    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:34.942111 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:34.942082    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6s2\" (UniqueName: \"kubernetes.io/projected/7070d490-016f-478a-a093-778cc0eb45bd-kube-api-access-fb6s2\") pod \"migrator-8b4d697c5-p65wk\" (UID: \"7070d490-016f-478a-a093-778cc0eb45bd\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"
Sep 11 16:18:35.034882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.034804    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"
Sep 11 16:18:35.035053 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.035027    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.035097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.035061    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbqbz\" (UniqueName: \"kubernetes.io/projected/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-kube-api-access-vbqbz\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.035097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.035088    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-hub\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.035163 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.035150    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.035215 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.035205    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-ca\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.035248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.035224    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.036078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.036040    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.037814 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.037785    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-ca\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.037937 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.037829    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-hub\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.037937 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.037896    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.038048 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.037932    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.043104 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.043080    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbqbz\" (UniqueName: \"kubernetes.io/projected/a0d8bc91-67f7-4aa7-9019-0a1d53e42912-kube-api-access-vbqbz\") pod \"cluster-proxy-proxy-agent-7f54d5dcd7-cz649\" (UID: \"a0d8bc91-67f7-4aa7-9019-0a1d53e42912\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.145345 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.145315    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:18:35.237456 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.237426    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:35.237609 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:35.237593    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:35.237671 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:35.237660    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls podName:076fb9d4-54bd-41e5-9c42-8a9fb362020f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:51.237640801 +0000 UTC m=+106.193688423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ghwt8" (UID: "076fb9d4-54bd-41e5-9c42-8a9fb362020f") : secret "cluster-monitoring-operator-tls" not found
Sep 11 16:18:35.304308 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.304286    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk"]
Sep 11 16:18:35.306100 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:35.306077    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7070d490_016f_478a_a093_778cc0eb45bd.slice/crio-b769e559cc33e6af8a006ee686341fd8b3b74fe0ce035f93a1d607e6b06d086b WatchSource:0}: Error finding container b769e559cc33e6af8a006ee686341fd8b3b74fe0ce035f93a1d607e6b06d086b: Status 404 returned error can't find the container with id b769e559cc33e6af8a006ee686341fd8b3b74fe0ce035f93a1d607e6b06d086b
Sep 11 16:18:35.338124 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.338100    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:35.338200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.338137    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:35.338258 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:35.338246    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 16:18:35.338302 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:35.338298    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:51.338282381 +0000 UTC m=+106.294329992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : secret "router-metrics-certs-default" not found
Sep 11 16:18:35.338347 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:35.338314    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle podName:ba8f0c3a-d449-423a-94ff-5b2ca617c983 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:51.338305335 +0000 UTC m=+106.294352944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle") pod "router-default-78b74d9c58-fgt69" (UID: "ba8f0c3a-d449-423a-94ff-5b2ca617c983") : configmap references non-existent config key: service-ca.crt
Sep 11 16:18:35.414878 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.414852    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"]
Sep 11 16:18:35.418173 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:35.418151    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d8bc91_67f7_4aa7_9019_0a1d53e42912.slice/crio-ecfaab9b2522ea97247eb3573a6965a1dcafb2bf70109cff3571905ab8b71fca WatchSource:0}: Error finding container ecfaab9b2522ea97247eb3573a6965a1dcafb2bf70109cff3571905ab8b71fca: Status 404 returned error can't find the container with id ecfaab9b2522ea97247eb3573a6965a1dcafb2bf70109cff3571905ab8b71fca
Sep 11 16:18:35.866221 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.866178    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" event={"ID":"a0d8bc91-67f7-4aa7-9019-0a1d53e42912","Type":"ContainerStarted","Data":"ecfaab9b2522ea97247eb3573a6965a1dcafb2bf70109cff3571905ab8b71fca"}
Sep 11 16:18:35.868033 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:35.868006    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk" event={"ID":"7070d490-016f-478a-a093-778cc0eb45bd","Type":"ContainerStarted","Data":"b769e559cc33e6af8a006ee686341fd8b3b74fe0ce035f93a1d607e6b06d086b"}
Sep 11 16:18:37.876691 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:37.876655    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk" event={"ID":"7070d490-016f-478a-a093-778cc0eb45bd","Type":"ContainerStarted","Data":"7caddcac64b5df5e228e55ffb73db3c7f1b49dc905dde2afbe6e54f83a347f71"}
Sep 11 16:18:37.877100 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:37.876713    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk" event={"ID":"7070d490-016f-478a-a093-778cc0eb45bd","Type":"ContainerStarted","Data":"c02eaff85c0275483cacd8a18813cc7d050c2c3e956869a1739d44ba7809146b"}
Sep 11 16:18:37.901003 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:37.900943    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-p65wk" podStartSLOduration=1.858670373 podStartE2EDuration="3.90091889s" totalImagesPullingTime="2.042248517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:37.899751995 +0000 UTC m=+92.855799627" watchObservedRunningTime="2026-09-11 16:18:37.90091889 +0000 UTC m=+92.856966520"
Sep 11 16:18:38.883089 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:38.883054    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" event={"ID":"a0d8bc91-67f7-4aa7-9019-0a1d53e42912","Type":"ContainerStarted","Data":"a191385ec02e9c777739cdebe8aaf7936cddc4c4ca710eaba7e11a5690cfb438"}
Sep 11 16:18:38.883449 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:38.883098    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" event={"ID":"a0d8bc91-67f7-4aa7-9019-0a1d53e42912","Type":"ContainerStarted","Data":"e55eca2c3a4a794ee545fe25f3eb66ce39445f87a9506f114cae0f7698b2b711"}
Sep 11 16:18:38.883449 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:38.883110    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" event={"ID":"a0d8bc91-67f7-4aa7-9019-0a1d53e42912","Type":"ContainerStarted","Data":"50e9be338c5d9b22e54ff6a34c430a2845f1d4d504da9bed9e72deb8d43fec5d"}
Sep 11 16:18:38.917830 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:38.917787    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" podStartSLOduration=1.659137638 podStartE2EDuration="4.917772036s" totalImagesPullingTime="3.258634398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:38.917301476 +0000 UTC m=+93.873349104" watchObservedRunningTime="2026-09-11 16:18:38.917772036 +0000 UTC m=+93.873819665"
Sep 11 16:18:39.368753 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:39.368707    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:39.368912 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:39.368784    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:39.368912 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:39.368900    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:39.368998 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:39.368967    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls podName:9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:55.368953196 +0000 UTC m=+110.325000816 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9q95b" (UID: "9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:39.369305 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:39.369280    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:42.288868 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:42.288822    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:18:42.289246 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:42.288978    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:18:42.289246 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:42.289063    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls podName:61c0e3ee-7239-48a6-8823-82be2e360cae nodeName:}" failed. No retries permitted until 2026-09-11 16:19:46.289047541 +0000 UTC m=+161.245095160 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls") pod "dns-default-pgtfg" (UID: "61c0e3ee-7239-48a6-8823-82be2e360cae") : secret "dns-default-metrics-tls" not found
Sep 11 16:18:47.822791 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:47.822758    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bxrwt"
Sep 11 16:18:51.257791 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.257756    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:51.260091 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.260065    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/076fb9d4-54bd-41e5-9c42-8a9fb362020f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ghwt8\" (UID: \"076fb9d4-54bd-41e5-9c42-8a9fb362020f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:51.285938 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.285915    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"
Sep 11 16:18:51.358372 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.358347    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:51.358479 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.358381    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:51.359084 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.359064    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba8f0c3a-d449-423a-94ff-5b2ca617c983-service-ca-bundle\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:51.360613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.360592    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba8f0c3a-d449-423a-94ff-5b2ca617c983-metrics-certs\") pod \"router-default-78b74d9c58-fgt69\" (UID: \"ba8f0c3a-d449-423a-94ff-5b2ca617c983\") " pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:51.396768 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.396747    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:51.558361 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.558294    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8"]
Sep 11 16:18:51.560969 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:51.560934    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076fb9d4_54bd_41e5_9c42_8a9fb362020f.slice/crio-f47d18bc6b950b38cff2b682e8e1d49084b1a34f392291d82a658d247f2eb0e2 WatchSource:0}: Error finding container f47d18bc6b950b38cff2b682e8e1d49084b1a34f392291d82a658d247f2eb0e2: Status 404 returned error can't find the container with id f47d18bc6b950b38cff2b682e8e1d49084b1a34f392291d82a658d247f2eb0e2
Sep 11 16:18:51.669613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.669585    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78b74d9c58-fgt69"]
Sep 11 16:18:51.672670 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:51.672648    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8f0c3a_d449_423a_94ff_5b2ca617c983.slice/crio-c19c9ebae66b288e7b8644325c87e04927d9c61c309233dc5a7014aa71724f24 WatchSource:0}: Error finding container c19c9ebae66b288e7b8644325c87e04927d9c61c309233dc5a7014aa71724f24: Status 404 returned error can't find the container with id c19c9ebae66b288e7b8644325c87e04927d9c61c309233dc5a7014aa71724f24
Sep 11 16:18:51.926806 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.926773    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78b74d9c58-fgt69" event={"ID":"ba8f0c3a-d449-423a-94ff-5b2ca617c983","Type":"ContainerStarted","Data":"fddea471628cd2ea63efd754de7845dee3e652e59e60951a9a7346f17677f16a"}
Sep 11 16:18:51.926993 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.926825    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78b74d9c58-fgt69" event={"ID":"ba8f0c3a-d449-423a-94ff-5b2ca617c983","Type":"ContainerStarted","Data":"c19c9ebae66b288e7b8644325c87e04927d9c61c309233dc5a7014aa71724f24"}
Sep 11 16:18:51.927882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.927860    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8" event={"ID":"076fb9d4-54bd-41e5-9c42-8a9fb362020f","Type":"ContainerStarted","Data":"f47d18bc6b950b38cff2b682e8e1d49084b1a34f392291d82a658d247f2eb0e2"}
Sep 11 16:18:51.946851 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:51.946806    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78b74d9c58-fgt69" podStartSLOduration=32.946791946 podStartE2EDuration="32.946791946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:51.946037489 +0000 UTC m=+106.902085116" watchObservedRunningTime="2026-09-11 16:18:51.946791946 +0000 UTC m=+106.902839574"
Sep 11 16:18:52.397574 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:52.397525    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:52.400181 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:52.400160    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:52.931444 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:52.931405    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:52.932862 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:52.932841    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78b74d9c58-fgt69"
Sep 11 16:18:53.935751 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:53.935709    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8" event={"ID":"076fb9d4-54bd-41e5-9c42-8a9fb362020f","Type":"ContainerStarted","Data":"2c82ce7a98377df801e41981818ca294bf64a51a195ca2ea3c4920827618b803"}
Sep 11 16:18:53.952673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:53.952623    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ghwt8" podStartSLOduration=33.466198344 podStartE2EDuration="34.952608473s" totalImagesPullingTime="1.486410129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:53.951949144 +0000 UTC m=+108.907996772" watchObservedRunningTime="2026-09-11 16:18:53.952608473 +0000 UTC m=+108.908656101"
Sep 11 16:18:55.386427 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:55.386386    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:55.388471 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:55.388453    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9q95b\" (UID: \"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1\") " pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:55.661357 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:55.661293    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9q95b"
Sep 11 16:18:55.929029 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:55.928964    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9q95b"]
Sep 11 16:18:55.932436 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:18:55.932408    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2e4cf6_8d6a_4f6f_a59e_d8578917d1b1.slice/crio-f7845cf9e6d4e8955b652cffc4e1c9f92ac8d3eabe6a5ac260f90028d4080da2 WatchSource:0}: Error finding container f7845cf9e6d4e8955b652cffc4e1c9f92ac8d3eabe6a5ac260f90028d4080da2: Status 404 returned error can't find the container with id f7845cf9e6d4e8955b652cffc4e1c9f92ac8d3eabe6a5ac260f90028d4080da2
Sep 11 16:18:55.942801 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:55.942777    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9q95b" event={"ID":"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1","Type":"ContainerStarted","Data":"f7845cf9e6d4e8955b652cffc4e1c9f92ac8d3eabe6a5ac260f90028d4080da2"}
Sep 11 16:18:56.950947 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:56.950909    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9q95b" event={"ID":"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1","Type":"ContainerStarted","Data":"94196e60b42e820e95c257736de596eb87c0a021e9d9cf90b48815cfe4c17815"}
Sep 11 16:18:56.951307 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:56.951004    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9q95b" event={"ID":"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1","Type":"ContainerStarted","Data":"273bb3ca53a88ec8430f8278651464b6d7a0b10866fbfdc14a239c2d8ecb33eb"}
Sep 11 16:18:57.955755 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:57.955730    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9q95b" event={"ID":"9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1","Type":"ContainerStarted","Data":"c72d5e70ff837c4569b64bea1a0a0a1e6241573c6360b0cc11dbfda44470c069"}
Sep 11 16:18:57.973764 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:57.973717    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9q95b" podStartSLOduration=33.141803099 podStartE2EDuration="34.97370072s" totalImagesPullingTime="1.831897621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:57.972760979 +0000 UTC m=+112.928808606" watchObservedRunningTime="2026-09-11 16:18:57.97370072 +0000 UTC m=+112.929748366"
Sep 11 16:18:58.733981 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.733948    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65fff7c679-txv88"]
Sep 11 16:18:58.735984 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.735967    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.742627 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.742604    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 16:18:58.742627 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.742619    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 16:18:58.742880 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.742867    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nwnk6\""
Sep 11 16:18:58.743107 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.743091    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 16:18:58.756422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.756400    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 16:18:58.757722 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.757701    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65fff7c679-txv88"]
Sep 11 16:18:58.822244 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.822221    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65fff7c679-txv88"]
Sep 11 16:18:58.822391 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:18:58.822373    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-fsm7m registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-fsm7m registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-65fff7c679-txv88" podUID="4dbf577a-d3ca-442e-95b5-91548b0b084d"
Sep 11 16:18:58.909895 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.909839    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-trusted-ca\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.909895 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.909867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dbf577a-d3ca-442e-95b5-91548b0b084d-ca-trust-extracted\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.909895 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.909886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-certificates\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.910043 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.909969    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-installation-pull-secrets\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.910043 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.910030    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-bound-sa-token\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.910116 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.910048    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsm7m\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-kube-api-access-fsm7m\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.910116 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.910067    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-tls\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.910116 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.910095    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-image-registry-private-configuration\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.958421 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.958394    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:58.963630 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:58.963612    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011086 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011059    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-bound-sa-token\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011086 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011086    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsm7m\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-kube-api-access-fsm7m\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011108    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-tls\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011129    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-image-registry-private-configuration\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011156    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-trusted-ca\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011182    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dbf577a-d3ca-442e-95b5-91548b0b084d-ca-trust-extracted\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011247    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-certificates\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-installation-pull-secrets\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.011630 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.011608    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dbf577a-d3ca-442e-95b5-91548b0b084d-ca-trust-extracted\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.012136 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.012116    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-certificates\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.012268 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.012242    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-trusted-ca\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.013714 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.013688    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-tls\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.013806 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.013719    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-installation-pull-secrets\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.013806 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.013751    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-image-registry-private-configuration\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.023119 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.023100    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-bound-sa-token\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.024117 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.024097    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsm7m\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-kube-api-access-fsm7m\") pod \"image-registry-65fff7c679-txv88\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") " pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.111672 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111651    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-certificates\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.111761 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111677    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dbf577a-d3ca-442e-95b5-91548b0b084d-ca-trust-extracted\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.111761 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111693    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-installation-pull-secrets\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.111761 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111724    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-trusted-ca\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.111909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111766    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-bound-sa-token\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.111909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111804    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-tls\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.111909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111844    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-image-registry-private-configuration\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.112053 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111925    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbf577a-d3ca-442e-95b5-91548b0b084d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:18:59.112053 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.111983    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsm7m\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-kube-api-access-fsm7m\") pod \"4dbf577a-d3ca-442e-95b5-91548b0b084d\" (UID: \"4dbf577a-d3ca-442e-95b5-91548b0b084d\") "
Sep 11 16:18:59.112160 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.112066    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:59.112245 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.112219    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:59.112298 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.112279    2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-certificates\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.112350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.112297    2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4dbf577a-d3ca-442e-95b5-91548b0b084d-ca-trust-extracted\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.112350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.112312    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4dbf577a-d3ca-442e-95b5-91548b0b084d-trusted-ca\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.113731 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.113699    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:59.113830 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.113810    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:59.114040 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.114022    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:59.114164 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.114146    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:59.114250 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.114229    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-kube-api-access-fsm7m" (OuterVolumeSpecName: "kube-api-access-fsm7m") pod "4dbf577a-d3ca-442e-95b5-91548b0b084d" (UID: "4dbf577a-d3ca-442e-95b5-91548b0b084d"). InnerVolumeSpecName "kube-api-access-fsm7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:59.213137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.213080    2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-registry-tls\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.213137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.213103    2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-image-registry-private-configuration\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.213137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.213116    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsm7m\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-kube-api-access-fsm7m\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.213137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.213128    2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4dbf577a-d3ca-442e-95b5-91548b0b084d-installation-pull-secrets\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.213308 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.213141    2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4dbf577a-d3ca-442e-95b5-91548b0b084d-bound-sa-token\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:18:59.961007 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.960979    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65fff7c679-txv88"
Sep 11 16:18:59.995019 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:18:59.994986    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65fff7c679-txv88"]
Sep 11 16:19:00.000935 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:00.000890    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65fff7c679-txv88"]
Sep 11 16:19:01.475299 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:01.475263    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dbf577a-d3ca-442e-95b5-91548b0b084d" path="/var/lib/kubelet/pods/4dbf577a-d3ca-442e-95b5-91548b0b084d/volumes"
Sep 11 16:19:06.143101 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.143069    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bzx5d"]
Sep 11 16:19:06.145742 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.145722    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.148521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.148499    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2bzxn\""
Sep 11 16:19:06.148521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.148509    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:19:06.148709 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.148509    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:19:06.148709 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.148566    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:19:06.148862 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.148847    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:19:06.253876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.253849    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-tls\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.253888    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.253905    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t654j\" (UniqueName: \"kubernetes.io/projected/c5f70df6-5a80-495c-a29e-c18db7697bf3-kube-api-access-t654j\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.253933    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-sys\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.253988    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-root\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.254007    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.254061    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5f70df6-5a80-495c-a29e-c18db7697bf3-metrics-client-ca\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254311 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.254102    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-textfile\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.254311 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.254122    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-wtmp\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.354849 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354813    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-root\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.354849 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354853    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354873    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5f70df6-5a80-495c-a29e-c18db7697bf3-metrics-client-ca\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-textfile\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354924    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-wtmp\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354928    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-root\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.354989    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-tls\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355052 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355042    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355067    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t654j\" (UniqueName: \"kubernetes.io/projected/c5f70df6-5a80-495c-a29e-c18db7697bf3-kube-api-access-t654j\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355082    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-wtmp\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355098    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-sys\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355177    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5f70df6-5a80-495c-a29e-c18db7697bf3-sys\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355499 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355340    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-textfile\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355766 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355743    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5f70df6-5a80-495c-a29e-c18db7697bf3-metrics-client-ca\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.355838 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.355784    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-accelerators-collector-config\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.357235 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.357207    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-tls\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.357343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.357242    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5f70df6-5a80-495c-a29e-c18db7697bf3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.363690 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.363671    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t654j\" (UniqueName: \"kubernetes.io/projected/c5f70df6-5a80-495c-a29e-c18db7697bf3-kube-api-access-t654j\") pod \"node-exporter-bzx5d\" (UID: \"c5f70df6-5a80-495c-a29e-c18db7697bf3\") " pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.453896 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.453848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bzx5d"
Sep 11 16:19:06.462708 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:19:06.462688    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f70df6_5a80_495c_a29e_c18db7697bf3.slice/crio-eb95286976fcd6776657d934ebd3015a6e436dbe3c04ae610064b4f9a34fb19e WatchSource:0}: Error finding container eb95286976fcd6776657d934ebd3015a6e436dbe3c04ae610064b4f9a34fb19e: Status 404 returned error can't find the container with id eb95286976fcd6776657d934ebd3015a6e436dbe3c04ae610064b4f9a34fb19e
Sep 11 16:19:06.983180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:06.983101    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzx5d" event={"ID":"c5f70df6-5a80-495c-a29e-c18db7697bf3","Type":"ContainerStarted","Data":"eb95286976fcd6776657d934ebd3015a6e436dbe3c04ae610064b4f9a34fb19e"}
Sep 11 16:19:07.987350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:07.987318    2550 generic.go:356] "Generic (PLEG): container finished" podID="c5f70df6-5a80-495c-a29e-c18db7697bf3" containerID="3b8a64ff2926293febfb3f2ce782286cd3fb7a6c43d8af60292b54456e3c799b" exitCode=0
Sep 11 16:19:07.987809 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:07.987383    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzx5d" event={"ID":"c5f70df6-5a80-495c-a29e-c18db7697bf3","Type":"ContainerDied","Data":"3b8a64ff2926293febfb3f2ce782286cd3fb7a6c43d8af60292b54456e3c799b"}
Sep 11 16:19:08.992636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:08.992604    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzx5d" event={"ID":"c5f70df6-5a80-495c-a29e-c18db7697bf3","Type":"ContainerStarted","Data":"144cf58c0b5e5daf2dcaf9e447d933be707b3fa7a7533732b618be700baee006"}
Sep 11 16:19:08.992986 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:08.992657    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzx5d" event={"ID":"c5f70df6-5a80-495c-a29e-c18db7697bf3","Type":"ContainerStarted","Data":"5aec189078986007954f55bb379829a94cf704b9b655fb71949f1cf167d9a0ae"}
Sep 11 16:19:09.015067 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:09.015001    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bzx5d" podStartSLOduration=2.094344922 podStartE2EDuration="3.014975072s" totalImagesPullingTime="860.850076ms" totalInitContainerRuntime="59.780074ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:09.013291445 +0000 UTC m=+123.969339074" watchObservedRunningTime="2026-09-11 16:19:09.014975072 +0000 UTC m=+123.971022700"
Sep 11 16:19:15.322779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:15.322746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:19:15.325093 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:15.325068    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac1c7b7-7288-434a-8196-cd1a17d2f0b8-metrics-certs\") pod \"network-metrics-daemon-vcxfz\" (UID: \"dac1c7b7-7288-434a-8196-cd1a17d2f0b8\") " pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:19:15.497792 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:15.497765    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks2m\""
Sep 11 16:19:15.506083 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:15.506064    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcxfz"
Sep 11 16:19:15.773739 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:15.773709    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vcxfz"]
Sep 11 16:19:15.776842 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:19:15.776811    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac1c7b7_7288_434a_8196_cd1a17d2f0b8.slice/crio-65fee90f0186d6dd07dc31f2cc620bd55d96a80fdb8fe93271edd3c98cae4118 WatchSource:0}: Error finding container 65fee90f0186d6dd07dc31f2cc620bd55d96a80fdb8fe93271edd3c98cae4118: Status 404 returned error can't find the container with id 65fee90f0186d6dd07dc31f2cc620bd55d96a80fdb8fe93271edd3c98cae4118
Sep 11 16:19:16.015502 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:16.015461    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcxfz" event={"ID":"dac1c7b7-7288-434a-8196-cd1a17d2f0b8","Type":"ContainerStarted","Data":"65fee90f0186d6dd07dc31f2cc620bd55d96a80fdb8fe93271edd3c98cae4118"}
Sep 11 16:19:17.020588 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:17.020499    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcxfz" event={"ID":"dac1c7b7-7288-434a-8196-cd1a17d2f0b8","Type":"ContainerStarted","Data":"f53782086533fbac4dc117634c20855dd05bf3068b19a4520336395dbfdaf723"}
Sep 11 16:19:17.020588 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:17.020577    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcxfz" event={"ID":"dac1c7b7-7288-434a-8196-cd1a17d2f0b8","Type":"ContainerStarted","Data":"42a349338706abbc5797df74e69bab3efa243322dab83e07e15b47937f8bd7b1"}
Sep 11 16:19:17.038237 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:17.038183    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vcxfz" podStartSLOduration=130.200982363 podStartE2EDuration="2m11.038169027s" totalImagesPullingTime="837.186664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:17.037688137 +0000 UTC m=+131.993735766" watchObservedRunningTime="2026-09-11 16:19:17.038169027 +0000 UTC m=+131.994216646"
Sep 11 16:19:28.513990 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.513960    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-qtjs4"]
Sep 11 16:19:28.516846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.516820    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:28.518861 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.518837    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 16:19:28.518996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.518879    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 16:19:28.519114 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.519098    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lktv4\""
Sep 11 16:19:28.533958 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.533934    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qtjs4"]
Sep 11 16:19:28.607875 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.607845    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lr9x\" (UniqueName: \"kubernetes.io/projected/4d6a0e34-05fa-4bd7-beba-1caf4426e69f-kube-api-access-7lr9x\") pod \"downloads-76f99584f4-qtjs4\" (UID: \"4d6a0e34-05fa-4bd7-beba-1caf4426e69f\") " pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:28.708470 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.708441    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lr9x\" (UniqueName: \"kubernetes.io/projected/4d6a0e34-05fa-4bd7-beba-1caf4426e69f-kube-api-access-7lr9x\") pod \"downloads-76f99584f4-qtjs4\" (UID: \"4d6a0e34-05fa-4bd7-beba-1caf4426e69f\") " pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:28.717329 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.717308    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lr9x\" (UniqueName: \"kubernetes.io/projected/4d6a0e34-05fa-4bd7-beba-1caf4426e69f-kube-api-access-7lr9x\") pod \"downloads-76f99584f4-qtjs4\" (UID: \"4d6a0e34-05fa-4bd7-beba-1caf4426e69f\") " pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:28.826684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:28.826628    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:29.098761 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:29.098729    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qtjs4"]
Sep 11 16:19:29.102274 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:19:29.102245    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6a0e34_05fa_4bd7_beba_1caf4426e69f.slice/crio-c339b14785ca190f3e6e6b0c9669ca01b3c9953bcec3063a790fa753dcd5e7cc WatchSource:0}: Error finding container c339b14785ca190f3e6e6b0c9669ca01b3c9953bcec3063a790fa753dcd5e7cc: Status 404 returned error can't find the container with id c339b14785ca190f3e6e6b0c9669ca01b3c9953bcec3063a790fa753dcd5e7cc
Sep 11 16:19:30.066624 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:30.066562    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qtjs4" event={"ID":"4d6a0e34-05fa-4bd7-beba-1caf4426e69f","Type":"ContainerStarted","Data":"c339b14785ca190f3e6e6b0c9669ca01b3c9953bcec3063a790fa753dcd5e7cc"}
Sep 11 16:19:41.400662 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:19:41.400615    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pgtfg" podUID="61c0e3ee-7239-48a6-8823-82be2e360cae"
Sep 11 16:19:42.110373 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:42.110336    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:19:45.123462 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:45.123406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qtjs4" event={"ID":"4d6a0e34-05fa-4bd7-beba-1caf4426e69f","Type":"ContainerStarted","Data":"746f46d736e0f25c866725746c7e523763d8dea1ef8211215e5af72aea1c8dbc"}
Sep 11 16:19:45.123942 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:45.123589    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:45.137618 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:45.137585    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-qtjs4"
Sep 11 16:19:45.146428 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:45.146376    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-qtjs4" podStartSLOduration=1.609089484 podStartE2EDuration="17.146357158s" totalImagesPullingTime="15.537267674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:45.145776718 +0000 UTC m=+160.101824346" watchObservedRunningTime="2026-09-11 16:19:45.146357158 +0000 UTC m=+160.102404790"
Sep 11 16:19:46.357745 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:46.357691    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:19:46.360432 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:46.360407    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/61c0e3ee-7239-48a6-8823-82be2e360cae-metrics-tls\") pod \"dns-default-pgtfg\" (UID: \"61c0e3ee-7239-48a6-8823-82be2e360cae\") " pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:19:46.614029 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:46.613941    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-244pg\""
Sep 11 16:19:46.622033 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:46.622003    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:19:46.903625 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:46.903485    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pgtfg"]
Sep 11 16:19:46.906422 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:19:46.906385    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c0e3ee_7239_48a6_8823_82be2e360cae.slice/crio-e8ecc9ea77fb9e0683737c6f61c41e6b67044a3897ed8d9d950c3355dc3fa6ff WatchSource:0}: Error finding container e8ecc9ea77fb9e0683737c6f61c41e6b67044a3897ed8d9d950c3355dc3fa6ff: Status 404 returned error can't find the container with id e8ecc9ea77fb9e0683737c6f61c41e6b67044a3897ed8d9d950c3355dc3fa6ff
Sep 11 16:19:47.132087 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:47.132042    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgtfg" event={"ID":"61c0e3ee-7239-48a6-8823-82be2e360cae","Type":"ContainerStarted","Data":"e8ecc9ea77fb9e0683737c6f61c41e6b67044a3897ed8d9d950c3355dc3fa6ff"}
Sep 11 16:19:50.145639 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:50.145599    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgtfg" event={"ID":"61c0e3ee-7239-48a6-8823-82be2e360cae","Type":"ContainerStarted","Data":"e363e2bb5a82a01923aaa1a14d5232524b673027b2f2a1f4ce66220d06f09f97"}
Sep 11 16:19:50.146068 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:50.145660    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pgtfg" event={"ID":"61c0e3ee-7239-48a6-8823-82be2e360cae","Type":"ContainerStarted","Data":"c79fb04a9e0cd0cda48f09dd2f62aaec7aec367c1d5f8a62b3fbb283fbc3bd6f"}
Sep 11 16:19:50.146068 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:50.145750    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:19:50.168258 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:19:50.168203    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pgtfg" podStartSLOduration=129.892029459 podStartE2EDuration="2m12.16818735s" totalImagesPullingTime="2.276157891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:50.167604243 +0000 UTC m=+165.123651871" watchObservedRunningTime="2026-09-11 16:19:50.16818735 +0000 UTC m=+165.124234978"
Sep 11 16:20:00.151319 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:00.151239    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pgtfg"
Sep 11 16:20:15.146403 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:15.146357    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" podUID="a0d8bc91-67f7-4aa7-9019-0a1d53e42912" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 16:20:25.147258 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:25.147218    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" podUID="a0d8bc91-67f7-4aa7-9019-0a1d53e42912" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 16:20:35.146465 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.146426    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" podUID="a0d8bc91-67f7-4aa7-9019-0a1d53e42912" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 16:20:35.146867 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.146494    2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649"
Sep 11 16:20:35.146984 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.146951    2550 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a191385ec02e9c777739cdebe8aaf7936cddc4c4ca710eaba7e11a5690cfb438"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 16:20:35.147021 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.147006    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" podUID="a0d8bc91-67f7-4aa7-9019-0a1d53e42912" containerName="service-proxy" containerID="cri-o://a191385ec02e9c777739cdebe8aaf7936cddc4c4ca710eaba7e11a5690cfb438" gracePeriod=30
Sep 11 16:20:35.306138 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.306110    2550 generic.go:356] "Generic (PLEG): container finished" podID="a0d8bc91-67f7-4aa7-9019-0a1d53e42912" containerID="a191385ec02e9c777739cdebe8aaf7936cddc4c4ca710eaba7e11a5690cfb438" exitCode=2
Sep 11 16:20:35.306241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.306162    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" event={"ID":"a0d8bc91-67f7-4aa7-9019-0a1d53e42912","Type":"ContainerDied","Data":"a191385ec02e9c777739cdebe8aaf7936cddc4c4ca710eaba7e11a5690cfb438"}
Sep 11 16:20:35.306241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:20:35.306203    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f54d5dcd7-cz649" event={"ID":"a0d8bc91-67f7-4aa7-9019-0a1d53e42912","Type":"ContainerStarted","Data":"770c0fdf50ff3f173704de22ed2aa67b95f1cc1a12b359a7b585635e60eef416"}
Sep 11 16:22:05.399831 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:22:05.399799    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:22:05.400495 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:22:05.400467    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:22:05.404558 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:22:05.404513    2550 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:24:38.536284 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.536205    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"]
Sep 11 16:24:38.538286 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.538269    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.540226 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.540201    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 16:24:38.540360 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.540234    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 16:24:38.540459 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.540447    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gfpz\""
Sep 11 16:24:38.553258 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.553237    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"]
Sep 11 16:24:38.580083 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.580059    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.580192 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.580090    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6qb\" (UniqueName: \"kubernetes.io/projected/d3c12098-aed2-4743-a8b5-5de4ca74f59c-kube-api-access-mw6qb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.580192 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.580171    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.681359 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.681318    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6qb\" (UniqueName: \"kubernetes.io/projected/d3c12098-aed2-4743-a8b5-5de4ca74f59c-kube-api-access-mw6qb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.681504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.681390    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.681504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.681415    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.681752 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.681736    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.681813 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.681770    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.689762 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.689742    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6qb\" (UniqueName: \"kubernetes.io/projected/d3c12098-aed2-4743-a8b5-5de4ca74f59c-kube-api-access-mw6qb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:38.846859 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:38.846833    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:39.112711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:39.112634    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"]
Sep 11 16:24:39.115457 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:24:39.115427    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c12098_aed2_4743_a8b5_5de4ca74f59c.slice/crio-63caa8ec34823b3c22b6a04ba21c7fdcdcd937de06e1567c13d626a6be0a39b9 WatchSource:0}: Error finding container 63caa8ec34823b3c22b6a04ba21c7fdcdcd937de06e1567c13d626a6be0a39b9: Status 404 returned error can't find the container with id 63caa8ec34823b3c22b6a04ba21c7fdcdcd937de06e1567c13d626a6be0a39b9
Sep 11 16:24:39.117455 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:39.117436    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:24:40.098584 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:40.098526    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8" event={"ID":"d3c12098-aed2-4743-a8b5-5de4ca74f59c","Type":"ContainerStarted","Data":"63caa8ec34823b3c22b6a04ba21c7fdcdcd937de06e1567c13d626a6be0a39b9"}
Sep 11 16:24:47.125070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:47.125038    2550 generic.go:356] "Generic (PLEG): container finished" podID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerID="918ace1ee61219ac48d2e8e77a16f0cbbaff4cc56e3d190fa4a259219ee68072" exitCode=0
Sep 11 16:24:47.125393 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:47.125134    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8" event={"ID":"d3c12098-aed2-4743-a8b5-5de4ca74f59c","Type":"ContainerDied","Data":"918ace1ee61219ac48d2e8e77a16f0cbbaff4cc56e3d190fa4a259219ee68072"}
Sep 11 16:24:49.134571 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:49.134527    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8" event={"ID":"d3c12098-aed2-4743-a8b5-5de4ca74f59c","Type":"ContainerStarted","Data":"89f33c3d1ad2923889e7386d09c937f19cff0719ba03c31d2d1c052129b5288e"}
Sep 11 16:24:50.139973 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:50.139940    2550 generic.go:356] "Generic (PLEG): container finished" podID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerID="89f33c3d1ad2923889e7386d09c937f19cff0719ba03c31d2d1c052129b5288e" exitCode=0
Sep 11 16:24:50.140383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:50.140030    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8" event={"ID":"d3c12098-aed2-4743-a8b5-5de4ca74f59c","Type":"ContainerDied","Data":"89f33c3d1ad2923889e7386d09c937f19cff0719ba03c31d2d1c052129b5288e"}
Sep 11 16:24:57.168357 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:57.168326    2550 generic.go:356] "Generic (PLEG): container finished" podID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerID="be6dd98e337c816c2c83ad2a86624cd1efc4053b5d855de458fab7862f187255" exitCode=0
Sep 11 16:24:57.168357 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:57.168361    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8" event={"ID":"d3c12098-aed2-4743-a8b5-5de4ca74f59c","Type":"ContainerDied","Data":"be6dd98e337c816c2c83ad2a86624cd1efc4053b5d855de458fab7862f187255"}
Sep 11 16:24:58.288762 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.288732    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:58.425634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.425575    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw6qb\" (UniqueName: \"kubernetes.io/projected/d3c12098-aed2-4743-a8b5-5de4ca74f59c-kube-api-access-mw6qb\") pod \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") "
Sep 11 16:24:58.425634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.425626    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-util\") pod \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") "
Sep 11 16:24:58.425818 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.425646    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-bundle\") pod \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\" (UID: \"d3c12098-aed2-4743-a8b5-5de4ca74f59c\") "
Sep 11 16:24:58.426265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.426240    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-bundle" (OuterVolumeSpecName: "bundle") pod "d3c12098-aed2-4743-a8b5-5de4ca74f59c" (UID: "d3c12098-aed2-4743-a8b5-5de4ca74f59c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:24:58.427635 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.427600    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c12098-aed2-4743-a8b5-5de4ca74f59c-kube-api-access-mw6qb" (OuterVolumeSpecName: "kube-api-access-mw6qb") pod "d3c12098-aed2-4743-a8b5-5de4ca74f59c" (UID: "d3c12098-aed2-4743-a8b5-5de4ca74f59c"). InnerVolumeSpecName "kube-api-access-mw6qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:24:58.431099 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.431074    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-util" (OuterVolumeSpecName: "util") pod "d3c12098-aed2-4743-a8b5-5de4ca74f59c" (UID: "d3c12098-aed2-4743-a8b5-5de4ca74f59c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:24:58.526645 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.526620    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:24:58.526645 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.526642    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c12098-aed2-4743-a8b5-5de4ca74f59c-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:24:58.526775 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:58.526651    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw6qb\" (UniqueName: \"kubernetes.io/projected/d3c12098-aed2-4743-a8b5-5de4ca74f59c-kube-api-access-mw6qb\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:24:59.176053 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:59.176019    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8" event={"ID":"d3c12098-aed2-4743-a8b5-5de4ca74f59c","Type":"ContainerDied","Data":"63caa8ec34823b3c22b6a04ba21c7fdcdcd937de06e1567c13d626a6be0a39b9"}
Sep 11 16:24:59.176218 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:59.176059    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1anhbr8"
Sep 11 16:24:59.176218 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:24:59.176065    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63caa8ec34823b3c22b6a04ba21c7fdcdcd937de06e1567c13d626a6be0a39b9"
Sep 11 16:25:10.792711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792680    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-772f7"]
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792912    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="pull"
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792923    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="pull"
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792937    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="util"
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792943    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="util"
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792954    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="extract"
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792959    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="extract"
Sep 11 16:25:10.793169 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.792997    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c12098-aed2-4743-a8b5-5de4ca74f59c" containerName="extract"
Sep 11 16:25:10.796942 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.796926    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:10.798855 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.798829    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 11 16:25:10.798997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.798889    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 11 16:25:10.798997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.798898    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-h8278\""
Sep 11 16:25:10.798997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.798900    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 11 16:25:10.798997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.798928    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 11 16:25:10.798997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.798917    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 11 16:25:10.804175 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.804152    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-772f7"]
Sep 11 16:25:10.910968 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.910937    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgjm\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-kube-api-access-6bgjm\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:10.911112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.911017    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:10.911112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:10.911042    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-cabundle0\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.003252 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.003223    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"]
Sep 11 16:25:11.004999 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.004985    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.007807 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.007789    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 11 16:25:11.011332 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.011312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.011422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.011336    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-cabundle0\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.011422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.011361    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgjm\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-kube-api-access-6bgjm\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.011590 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.011574    2550 secret.go:281] references non-existent secret key: ca.crt
Sep 11 16:25:11.011663 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.011592    2550 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 11 16:25:11.011663 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.011600    2550 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-772f7: references non-existent secret key: ca.crt
Sep 11 16:25:11.011663 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.011658    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates podName:d369ecbe-ec98-4c38-b05c-fd9007ec9e4c nodeName:}" failed. No retries permitted until 2026-09-11 16:25:11.511643669 +0000 UTC m=+486.467691278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates") pod "keda-operator-579c8958f7-772f7" (UID: "d369ecbe-ec98-4c38-b05c-fd9007ec9e4c") : references non-existent secret key: ca.crt
Sep 11 16:25:11.011906 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.011888    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-cabundle0\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.025295 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.025274    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"]
Sep 11 16:25:11.029660 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.029638    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgjm\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-kube-api-access-6bgjm\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.112504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.112479    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e41409a0-d5b4-4d88-8f6d-363eed866b20-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.112617 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.112507    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjwd\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-kube-api-access-7bjwd\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.112617 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.112530    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.213207 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.213185    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e41409a0-d5b4-4d88-8f6d-363eed866b20-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.213300 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.213210    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjwd\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-kube-api-access-7bjwd\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.213300 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.213232    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.213367 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.213319    2550 secret.go:281] references non-existent secret key: tls.crt
Sep 11 16:25:11.213367 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.213331    2550 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 11 16:25:11.213367 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.213349    2550 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 11 16:25:11.213367 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.213365    2550 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 11 16:25:11.213510 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.213449    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates podName:e41409a0-d5b4-4d88-8f6d-363eed866b20 nodeName:}" failed. No retries permitted until 2026-09-11 16:25:11.713430989 +0000 UTC m=+486.669478600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates") pod "keda-metrics-apiserver-77857b7b98-rscwm" (UID: "e41409a0-d5b4-4d88-8f6d-363eed866b20") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 11 16:25:11.213609 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.213591    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e41409a0-d5b4-4d88-8f6d-363eed866b20-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.221091 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.221073    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjwd\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-kube-api-access-7bjwd\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.516030 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.515939    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:11.516193 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.516067    2550 secret.go:281] references non-existent secret key: ca.crt
Sep 11 16:25:11.516193 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.516084    2550 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 11 16:25:11.516193 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.516095    2550 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-772f7: references non-existent secret key: ca.crt
Sep 11 16:25:11.516193 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.516152    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates podName:d369ecbe-ec98-4c38-b05c-fd9007ec9e4c nodeName:}" failed. No retries permitted until 2026-09-11 16:25:12.516135603 +0000 UTC m=+487.472183213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates") pod "keda-operator-579c8958f7-772f7" (UID: "d369ecbe-ec98-4c38-b05c-fd9007ec9e4c") : references non-existent secret key: ca.crt
Sep 11 16:25:11.716938 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.716909    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:11.717097 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.717023    2550 secret.go:281] references non-existent secret key: tls.crt
Sep 11 16:25:11.717097 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.717037    2550 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 11 16:25:11.717097 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.717055    2550 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm: references non-existent secret key: tls.crt
Sep 11 16:25:11.717221 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:25:11.717105    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates podName:e41409a0-d5b4-4d88-8f6d-363eed866b20 nodeName:}" failed. No retries permitted until 2026-09-11 16:25:12.717092573 +0000 UTC m=+487.673140179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates") pod "keda-metrics-apiserver-77857b7b98-rscwm" (UID: "e41409a0-d5b4-4d88-8f6d-363eed866b20") : references non-existent secret key: tls.crt
Sep 11 16:25:11.769642 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.769572    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vd6ks"]
Sep 11 16:25:11.771739 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.771717    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:11.773391 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.773371    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 11 16:25:11.780375 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.780352    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vd6ks"]
Sep 11 16:25:11.917937 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.917911    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4096d570-8690-4bf0-a433-f053cd505612-certificates\") pod \"keda-admission-6c4d864b86-vd6ks\" (UID: \"4096d570-8690-4bf0-a433-f053cd505612\") " pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:11.918266 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:11.917961    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94hw\" (UniqueName: \"kubernetes.io/projected/4096d570-8690-4bf0-a433-f053cd505612-kube-api-access-t94hw\") pod \"keda-admission-6c4d864b86-vd6ks\" (UID: \"4096d570-8690-4bf0-a433-f053cd505612\") " pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:12.019173 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.019144    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94hw\" (UniqueName: \"kubernetes.io/projected/4096d570-8690-4bf0-a433-f053cd505612-kube-api-access-t94hw\") pod \"keda-admission-6c4d864b86-vd6ks\" (UID: \"4096d570-8690-4bf0-a433-f053cd505612\") " pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:12.019318 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.019213    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4096d570-8690-4bf0-a433-f053cd505612-certificates\") pod \"keda-admission-6c4d864b86-vd6ks\" (UID: \"4096d570-8690-4bf0-a433-f053cd505612\") " pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:12.021772 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.021726    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4096d570-8690-4bf0-a433-f053cd505612-certificates\") pod \"keda-admission-6c4d864b86-vd6ks\" (UID: \"4096d570-8690-4bf0-a433-f053cd505612\") " pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:12.026855 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.026832    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94hw\" (UniqueName: \"kubernetes.io/projected/4096d570-8690-4bf0-a433-f053cd505612-kube-api-access-t94hw\") pod \"keda-admission-6c4d864b86-vd6ks\" (UID: \"4096d570-8690-4bf0-a433-f053cd505612\") " pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:12.082623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.082598    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:12.350234 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.350207    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-vd6ks"]
Sep 11 16:25:12.355659 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:25:12.355607    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4096d570_8690_4bf0_a433_f053cd505612.slice/crio-03968b2b4b71490b10bc8cd79bc02904a017b133573ce60b89ae1aa120252e70 WatchSource:0}: Error finding container 03968b2b4b71490b10bc8cd79bc02904a017b133573ce60b89ae1aa120252e70: Status 404 returned error can't find the container with id 03968b2b4b71490b10bc8cd79bc02904a017b133573ce60b89ae1aa120252e70
Sep 11 16:25:12.524334 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.524311    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:12.526556 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.526515    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d369ecbe-ec98-4c38-b05c-fd9007ec9e4c-certificates\") pod \"keda-operator-579c8958f7-772f7\" (UID: \"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c\") " pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:12.607575 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.607495    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:12.725907 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.725876    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:12.728049 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.728028    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e41409a0-d5b4-4d88-8f6d-363eed866b20-certificates\") pod \"keda-metrics-apiserver-77857b7b98-rscwm\" (UID: \"e41409a0-d5b4-4d88-8f6d-363eed866b20\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:12.814129 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.814089    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:12.879846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:12.879783    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-772f7"]
Sep 11 16:25:12.882965 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:25:12.882939    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd369ecbe_ec98_4c38_b05c_fd9007ec9e4c.slice/crio-8e73e3c4416e62e7d49d7e3f0e5b7bc99e953eee2c8e281ac967bae7ca77351a WatchSource:0}: Error finding container 8e73e3c4416e62e7d49d7e3f0e5b7bc99e953eee2c8e281ac967bae7ca77351a: Status 404 returned error can't find the container with id 8e73e3c4416e62e7d49d7e3f0e5b7bc99e953eee2c8e281ac967bae7ca77351a
Sep 11 16:25:13.080226 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:13.080095    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"]
Sep 11 16:25:13.082630 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:25:13.082597    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41409a0_d5b4_4d88_8f6d_363eed866b20.slice/crio-b3449dd39357747700871f40ecaa7e8bbc0bc66f0e6d124bf76323dc4b068185 WatchSource:0}: Error finding container b3449dd39357747700871f40ecaa7e8bbc0bc66f0e6d124bf76323dc4b068185: Status 404 returned error can't find the container with id b3449dd39357747700871f40ecaa7e8bbc0bc66f0e6d124bf76323dc4b068185
Sep 11 16:25:13.224009 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:13.223935    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm" event={"ID":"e41409a0-d5b4-4d88-8f6d-363eed866b20","Type":"ContainerStarted","Data":"b3449dd39357747700871f40ecaa7e8bbc0bc66f0e6d124bf76323dc4b068185"}
Sep 11 16:25:13.225070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:13.225043    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-772f7" event={"ID":"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c","Type":"ContainerStarted","Data":"8e73e3c4416e62e7d49d7e3f0e5b7bc99e953eee2c8e281ac967bae7ca77351a"}
Sep 11 16:25:13.226102 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:13.226082    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks" event={"ID":"4096d570-8690-4bf0-a433-f053cd505612","Type":"ContainerStarted","Data":"03968b2b4b71490b10bc8cd79bc02904a017b133573ce60b89ae1aa120252e70"}
Sep 11 16:25:15.236008 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:15.235911    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks" event={"ID":"4096d570-8690-4bf0-a433-f053cd505612","Type":"ContainerStarted","Data":"adcef57f0036ef9b1d48d31f2480ca62d33912451959302a5522edc95c39c540"}
Sep 11 16:25:15.236403 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:15.236062    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:15.253976 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:15.253925    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks" podStartSLOduration=1.773609781 podStartE2EDuration="4.253911044s" totalImagesPullingTime="2.480301263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:15.252308794 +0000 UTC m=+490.208356423" watchObservedRunningTime="2026-09-11 16:25:15.253911044 +0000 UTC m=+490.209958672"
Sep 11 16:25:17.250601 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:17.250565    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm" event={"ID":"e41409a0-d5b4-4d88-8f6d-363eed866b20","Type":"ContainerStarted","Data":"c0ccdc707677db5937bbef9be541ff2a61bd5422db9b7134d8fc83bb67117bf0"}
Sep 11 16:25:17.251056 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:17.250687    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:17.252234 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:17.252207    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-772f7" event={"ID":"d369ecbe-ec98-4c38-b05c-fd9007ec9e4c","Type":"ContainerStarted","Data":"fc99ecc3a2a031ab6c72a41a3a05fd3443ee0578acd970d92f77f2f6f1bbe8b1"}
Sep 11 16:25:17.252343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:17.252266    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:25:17.279440 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:17.279396    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm" podStartSLOduration=3.586946353 podStartE2EDuration="7.279383258s" totalImagesPullingTime="3.692436905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:17.277801121 +0000 UTC m=+492.233848758" watchObservedRunningTime="2026-09-11 16:25:17.279383258 +0000 UTC m=+492.235430876"
Sep 11 16:25:17.312652 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:17.312611    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-772f7" podStartSLOduration=3.420449125 podStartE2EDuration="7.312594059s" totalImagesPullingTime="3.892144934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:17.310213938 +0000 UTC m=+492.266261565" watchObservedRunningTime="2026-09-11 16:25:17.312594059 +0000 UTC m=+492.268641690"
Sep 11 16:25:28.260685 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:28.260612    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-rscwm"
Sep 11 16:25:36.243106 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:36.243077    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-vd6ks"
Sep 11 16:25:38.258629 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:25:38.258600    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-772f7"
Sep 11 16:26:03.316090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.316000    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"]
Sep 11 16:26:03.318321 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.318298    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.322487 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.322465    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 16:26:03.322611 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.322569    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 16:26:03.322810 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.322794    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gfpz\""
Sep 11 16:26:03.328022 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.328000    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"]
Sep 11 16:26:03.378620 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.378592    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mqd9\" (UniqueName: \"kubernetes.io/projected/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-kube-api-access-6mqd9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.378710 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.378634    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.378710 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.378655    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.479109 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.479082    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mqd9\" (UniqueName: \"kubernetes.io/projected/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-kube-api-access-6mqd9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.479213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.479120    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.479213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.479140    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.479584 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.479564    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.479633 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.479570    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.487135 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.487114    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mqd9\" (UniqueName: \"kubernetes.io/projected/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-kube-api-access-6mqd9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.627088 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.627065    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:03.895642 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:03.895572    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"]
Sep 11 16:26:03.898191 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:26:03.898163    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b03d2b8_8a89_4c18_a8a6_2ca12d829f43.slice/crio-8d76570e408c53f0733fba82dd8cfb0101aa654ec59683a699bbfb64b3c4b7df WatchSource:0}: Error finding container 8d76570e408c53f0733fba82dd8cfb0101aa654ec59683a699bbfb64b3c4b7df: Status 404 returned error can't find the container with id 8d76570e408c53f0733fba82dd8cfb0101aa654ec59683a699bbfb64b3c4b7df
Sep 11 16:26:04.432960 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:04.432924    2550 generic.go:356] "Generic (PLEG): container finished" podID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerID="e34504aa1f76043a8e9f2f8603ead0a214e9cc813d4dd6a1427312c7ff9ad846" exitCode=0
Sep 11 16:26:04.433395 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:04.433004    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf" event={"ID":"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43","Type":"ContainerDied","Data":"e34504aa1f76043a8e9f2f8603ead0a214e9cc813d4dd6a1427312c7ff9ad846"}
Sep 11 16:26:04.433395 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:04.433060    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf" event={"ID":"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43","Type":"ContainerStarted","Data":"8d76570e408c53f0733fba82dd8cfb0101aa654ec59683a699bbfb64b3c4b7df"}
Sep 11 16:26:07.445812 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:07.445783    2550 generic.go:356] "Generic (PLEG): container finished" podID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerID="ee6f3f03e249922b827342bb02da7da77c7b10e80a88ec4967767e049a98b54a" exitCode=0
Sep 11 16:26:07.446245 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:07.445877    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf" event={"ID":"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43","Type":"ContainerDied","Data":"ee6f3f03e249922b827342bb02da7da77c7b10e80a88ec4967767e049a98b54a"}
Sep 11 16:26:08.451132 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:08.451091    2550 generic.go:356] "Generic (PLEG): container finished" podID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerID="aa7fc789ff1ec549a6a593beb68ccedd1afa6c1adb708fa9707d1e9395d516ca" exitCode=0
Sep 11 16:26:08.451518 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:08.451183    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf" event={"ID":"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43","Type":"ContainerDied","Data":"aa7fc789ff1ec549a6a593beb68ccedd1afa6c1adb708fa9707d1e9395d516ca"}
Sep 11 16:26:09.570223 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.570198    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:09.626155 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.626127    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-util\") pod \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") "
Sep 11 16:26:09.626265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.626173    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mqd9\" (UniqueName: \"kubernetes.io/projected/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-kube-api-access-6mqd9\") pod \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") "
Sep 11 16:26:09.626265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.626197    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-bundle\") pod \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\" (UID: \"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43\") "
Sep 11 16:26:09.626863 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.626837    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-bundle" (OuterVolumeSpecName: "bundle") pod "1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" (UID: "1b03d2b8-8a89-4c18-a8a6-2ca12d829f43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:09.628129 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.628105    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-kube-api-access-6mqd9" (OuterVolumeSpecName: "kube-api-access-6mqd9") pod "1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" (UID: "1b03d2b8-8a89-4c18-a8a6-2ca12d829f43"). InnerVolumeSpecName "kube-api-access-6mqd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:09.632211 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.632187    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-util" (OuterVolumeSpecName: "util") pod "1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" (UID: "1b03d2b8-8a89-4c18-a8a6-2ca12d829f43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:09.726973 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.726912    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:26:09.726973 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.726942    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mqd9\" (UniqueName: \"kubernetes.io/projected/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-kube-api-access-6mqd9\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:26:09.726973 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:09.726952    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b03d2b8-8a89-4c18-a8a6-2ca12d829f43-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:26:10.459956 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:10.459927    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf" event={"ID":"1b03d2b8-8a89-4c18-a8a6-2ca12d829f43","Type":"ContainerDied","Data":"8d76570e408c53f0733fba82dd8cfb0101aa654ec59683a699bbfb64b3c4b7df"}
Sep 11 16:26:10.460113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:10.459977    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d76570e408c53f0733fba82dd8cfb0101aa654ec59683a699bbfb64b3c4b7df"
Sep 11 16:26:10.460113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:10.459936    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ggvwf"
Sep 11 16:26:25.428289 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428248    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"]
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428661    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="util"
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428679    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="util"
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428694    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="pull"
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428702    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="pull"
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428715    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="extract"
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428722    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="extract"
Sep 11 16:26:25.428911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.428800    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b03d2b8-8a89-4c18-a8a6-2ca12d829f43" containerName="extract"
Sep 11 16:26:25.435908 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.435887    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.438376 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.438356    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 16:26:25.438491 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.438385    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gfpz\""
Sep 11 16:26:25.438758 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.438741    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 16:26:25.438961 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.438936    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"]
Sep 11 16:26:25.537102 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.537081    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.537209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.537118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.537209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.537145    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8hp\" (UniqueName: \"kubernetes.io/projected/4e06bc1d-5ca4-409d-beb5-676682fd9cda-kube-api-access-vb8hp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.637740 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.637716    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.637834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.637756    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.637834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.637791    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8hp\" (UniqueName: \"kubernetes.io/projected/4e06bc1d-5ca4-409d-beb5-676682fd9cda-kube-api-access-vb8hp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.638155 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.638136    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.638193 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.638141    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.645065 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.645042    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8hp\" (UniqueName: \"kubernetes.io/projected/4e06bc1d-5ca4-409d-beb5-676682fd9cda-kube-api-access-vb8hp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:25.745877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:25.745816    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:26.013884 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:26.013772    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"]
Sep 11 16:26:26.016380 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:26:26.016353    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e06bc1d_5ca4_409d_beb5_676682fd9cda.slice/crio-2940e6873c6f38756a2ef845f4fcc3e04471f839646d11b556200c85e918133a WatchSource:0}: Error finding container 2940e6873c6f38756a2ef845f4fcc3e04471f839646d11b556200c85e918133a: Status 404 returned error can't find the container with id 2940e6873c6f38756a2ef845f4fcc3e04471f839646d11b556200c85e918133a
Sep 11 16:26:26.517794 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:26.517760    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerID="2a45b4bfa3a328dda06eac1e5b7e13a3dc4e6727217a3f1f887eff814453c739" exitCode=0
Sep 11 16:26:26.518155 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:26.517847    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5" event={"ID":"4e06bc1d-5ca4-409d-beb5-676682fd9cda","Type":"ContainerDied","Data":"2a45b4bfa3a328dda06eac1e5b7e13a3dc4e6727217a3f1f887eff814453c739"}
Sep 11 16:26:26.518155 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:26.517895    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5" event={"ID":"4e06bc1d-5ca4-409d-beb5-676682fd9cda","Type":"ContainerStarted","Data":"2940e6873c6f38756a2ef845f4fcc3e04471f839646d11b556200c85e918133a"}
Sep 11 16:26:29.531143 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:29.531110    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerID="622aaf5a6e8872c2dbc518bfde518791c970c487141817061fff62fc8ed026d7" exitCode=0
Sep 11 16:26:29.531507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:29.531172    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5" event={"ID":"4e06bc1d-5ca4-409d-beb5-676682fd9cda","Type":"ContainerDied","Data":"622aaf5a6e8872c2dbc518bfde518791c970c487141817061fff62fc8ed026d7"}
Sep 11 16:26:30.536963 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:30.536926    2550 generic.go:356] "Generic (PLEG): container finished" podID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerID="53f2ed20cd78aac4fbf8069159d73e372e43fae499743ac9c56f76810db80236" exitCode=0
Sep 11 16:26:30.536963 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:30.536963    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5" event={"ID":"4e06bc1d-5ca4-409d-beb5-676682fd9cda","Type":"ContainerDied","Data":"53f2ed20cd78aac4fbf8069159d73e372e43fae499743ac9c56f76810db80236"}
Sep 11 16:26:31.660703 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.660681    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:31.780274 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.780249    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb8hp\" (UniqueName: \"kubernetes.io/projected/4e06bc1d-5ca4-409d-beb5-676682fd9cda-kube-api-access-vb8hp\") pod \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") "
Sep 11 16:26:31.780383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.780286    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-bundle\") pod \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") "
Sep 11 16:26:31.780383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.780301    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-util\") pod \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\" (UID: \"4e06bc1d-5ca4-409d-beb5-676682fd9cda\") "
Sep 11 16:26:31.780719 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.780695    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-bundle" (OuterVolumeSpecName: "bundle") pod "4e06bc1d-5ca4-409d-beb5-676682fd9cda" (UID: "4e06bc1d-5ca4-409d-beb5-676682fd9cda"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:31.782205 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.782176    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e06bc1d-5ca4-409d-beb5-676682fd9cda-kube-api-access-vb8hp" (OuterVolumeSpecName: "kube-api-access-vb8hp") pod "4e06bc1d-5ca4-409d-beb5-676682fd9cda" (UID: "4e06bc1d-5ca4-409d-beb5-676682fd9cda"). InnerVolumeSpecName "kube-api-access-vb8hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:31.785422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.785367    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-util" (OuterVolumeSpecName: "util") pod "4e06bc1d-5ca4-409d-beb5-676682fd9cda" (UID: "4e06bc1d-5ca4-409d-beb5-676682fd9cda"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:26:31.880843 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.880812    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8hp\" (UniqueName: \"kubernetes.io/projected/4e06bc1d-5ca4-409d-beb5-676682fd9cda-kube-api-access-vb8hp\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:26:31.880843 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.880843    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:26:31.880843 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:31.880852    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e06bc1d-5ca4-409d-beb5-676682fd9cda-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:26:32.547465 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:32.547440    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5"
Sep 11 16:26:32.547465 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:32.547462    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fm66l5" event={"ID":"4e06bc1d-5ca4-409d-beb5-676682fd9cda","Type":"ContainerDied","Data":"2940e6873c6f38756a2ef845f4fcc3e04471f839646d11b556200c85e918133a"}
Sep 11 16:26:32.547709 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:32.547501    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2940e6873c6f38756a2ef845f4fcc3e04471f839646d11b556200c85e918133a"
Sep 11 16:26:52.527182 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527149    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"]
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527392    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="pull"
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527403    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="pull"
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527416    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="util"
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527421    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="util"
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527434    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="extract"
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527439    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="extract"
Sep 11 16:26:52.527525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.527478    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e06bc1d-5ca4-409d-beb5-676682fd9cda" containerName="extract"
Sep 11 16:26:52.533717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.533685    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.535916 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.535885    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 16:26:52.535916 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.535904    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 16:26:52.536078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.535907    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mnqx7\""
Sep 11 16:26:52.536078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.535935    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 16:26:52.536078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.535949    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 16:26:52.543897 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.543876    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"]
Sep 11 16:26:52.613298 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613279    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.613298 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613304    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.613443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613335    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-kube-api-access-25sxl\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.613443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613389    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.613443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613426    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.613536 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8aa1c-ffad-420b-9f6f-a962c6519215-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.613536 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.613481    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714196 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714053    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8aa1c-ffad-420b-9f6f-a962c6519215-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714196 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714088    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714196 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714131    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714196 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714156    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714207    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-kube-api-access-25sxl\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714235    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.714971 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.714942    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.716231 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.716207    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8aa1c-ffad-420b-9f6f-a962c6519215-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.716231 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.716225    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.716422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.716401    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.716479 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.716428    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.722426 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.722401    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.722591 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.722568    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-kube-api-access-25sxl\") pod \"istiod-openshift-gateway-7b7995d5cd-xcfxf\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:52.842655 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:52.842638    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:26:53.114833 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:53.114717    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"]
Sep 11 16:26:53.117022 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:26:53.116989    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b8aa1c_ffad_420b_9f6f_a962c6519215.slice/crio-200cd2e0f2f4a964ba1fd60f34679fe20c2911fb0c64f6106cc1302cb71d76bb WatchSource:0}: Error finding container 200cd2e0f2f4a964ba1fd60f34679fe20c2911fb0c64f6106cc1302cb71d76bb: Status 404 returned error can't find the container with id 200cd2e0f2f4a964ba1fd60f34679fe20c2911fb0c64f6106cc1302cb71d76bb
Sep 11 16:26:53.625711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:53.625676    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" event={"ID":"99b8aa1c-ffad-420b-9f6f-a962c6519215","Type":"ContainerStarted","Data":"200cd2e0f2f4a964ba1fd60f34679fe20c2911fb0c64f6106cc1302cb71d76bb"}
Sep 11 16:26:59.657940 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:59.657902    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 16:26:59.658204 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:26:59.657975    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 16:27:00.653365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:00.653323    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" event={"ID":"99b8aa1c-ffad-420b-9f6f-a962c6519215","Type":"ContainerStarted","Data":"fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c"}
Sep 11 16:27:00.653535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:00.653506    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:27:00.655450 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:00.655419    2550 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xcfxf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 16:27:00.655587 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:00.655474    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" podUID="99b8aa1c-ffad-420b-9f6f-a962c6519215" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:27:00.673407 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:00.673358    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" podStartSLOduration=2.134505425 podStartE2EDuration="8.673339648s" totalImagesPullingTime="6.538834223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:00.671227103 +0000 UTC m=+595.627274733" watchObservedRunningTime="2026-09-11 16:27:00.673339648 +0000 UTC m=+595.629387277"
Sep 11 16:27:01.657675 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:01.657645    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:27:05.426224 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:05.426198    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:27:05.426696 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:05.426568    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:27:12.516139 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.516106    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"]
Sep 11 16:27:12.519348 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.519332    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.521113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.521086    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 16:27:12.521241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.521088    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 16:27:12.521452 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.521432    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5gfpz\""
Sep 11 16:27:12.526723 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.526698    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"]
Sep 11 16:27:12.617760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.617729    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"]
Sep 11 16:27:12.620958 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.620943    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.629985 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.629965    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"]
Sep 11 16:27:12.650590 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.650538    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxwc\" (UniqueName: \"kubernetes.io/projected/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-kube-api-access-cpxwc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.650715 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.650668    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.650777 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.650731    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.715207 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.715183    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"]
Sep 11 16:27:12.718363 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.718347    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.727044 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.727023    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"]
Sep 11 16:27:12.751631 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.751603    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxwc\" (UniqueName: \"kubernetes.io/projected/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-kube-api-access-cpxwc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.751726 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.751661    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkt8\" (UniqueName: \"kubernetes.io/projected/bf0baa69-0dc2-420f-be99-d18d8561fd26-kube-api-access-btkt8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.751726 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.751693    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.751726 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.751721    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.751839 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.751742    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.751881 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.751845    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.752028 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.752011    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.752112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.752096    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.758890 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.758868    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxwc\" (UniqueName: \"kubernetes.io/projected/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-kube-api-access-cpxwc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.814799 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.814742    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"]
Sep 11 16:27:12.818740 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.818723    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:12.827402 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.827372    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"]
Sep 11 16:27:12.830328 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.830295    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:12.852808 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.852780    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjkm\" (UniqueName: \"kubernetes.io/projected/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-kube-api-access-wtjkm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.852903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.852816    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkt8\" (UniqueName: \"kubernetes.io/projected/bf0baa69-0dc2-420f-be99-d18d8561fd26-kube-api-access-btkt8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.852903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.852851    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.852903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.852881    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.853059 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.853011    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.853059 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.853052    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.853242 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.853222    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.853314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.853252    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.860780 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.860755    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkt8\" (UniqueName: \"kubernetes.io/projected/bf0baa69-0dc2-420f-be99-d18d8561fd26-kube-api-access-btkt8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.929311 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.929283    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:12.953467 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.953467 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953472    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.953686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953495    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:12.953686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953517    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82bw\" (UniqueName: \"kubernetes.io/projected/a9ffc983-349c-4879-883c-a7fa79fa0d12-kube-api-access-k82bw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:12.953686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953583    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:12.953849 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953768    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjkm\" (UniqueName: \"kubernetes.io/projected/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-kube-api-access-wtjkm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.953939 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953920    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.954006 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.953943    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:12.961777 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:12.961753    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjkm\" (UniqueName: \"kubernetes.io/projected/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-kube-api-access-wtjkm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:13.027288 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.027244    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:13.054484 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.054460    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82bw\" (UniqueName: \"kubernetes.io/projected/a9ffc983-349c-4879-883c-a7fa79fa0d12-kube-api-access-k82bw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.054606 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.054491    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.054606 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.054578    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.054911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.054885    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.055003 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.054899    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.063357 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.063332    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82bw\" (UniqueName: \"kubernetes.io/projected/a9ffc983-349c-4879-883c-a7fa79fa0d12-kube-api-access-k82bw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.102678 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.102578    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"]
Sep 11 16:27:13.105270 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:13.105243    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf146af1_c5e8_40e9_8e09_7fcb14980ebe.slice/crio-0f504185fe4ff6f912fa4d66f9444b923b5d3843f9a37e223ea6989b9120cb7e WatchSource:0}: Error finding container 0f504185fe4ff6f912fa4d66f9444b923b5d3843f9a37e223ea6989b9120cb7e: Status 404 returned error can't find the container with id 0f504185fe4ff6f912fa4d66f9444b923b5d3843f9a37e223ea6989b9120cb7e
Sep 11 16:27:13.131266 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.131241    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:13.203110 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.203084    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"]
Sep 11 16:27:13.205975 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:13.205946    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0baa69_0dc2_420f_be99_d18d8561fd26.slice/crio-1c784a6121be1e7f93b3ceb10d9a529021035a39613b0ba8ca876c022a0467e9 WatchSource:0}: Error finding container 1c784a6121be1e7f93b3ceb10d9a529021035a39613b0ba8ca876c022a0467e9: Status 404 returned error can't find the container with id 1c784a6121be1e7f93b3ceb10d9a529021035a39613b0ba8ca876c022a0467e9
Sep 11 16:27:13.295112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.295085    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"]
Sep 11 16:27:13.298259 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:13.298232    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250bfaf5_a1f2_4ca3_8f49_480eaa824c27.slice/crio-802d30449c01617fe2d7d9feffc58e6ef915a5237bd51f58f52511b62464e772 WatchSource:0}: Error finding container 802d30449c01617fe2d7d9feffc58e6ef915a5237bd51f58f52511b62464e772: Status 404 returned error can't find the container with id 802d30449c01617fe2d7d9feffc58e6ef915a5237bd51f58f52511b62464e772
Sep 11 16:27:13.401694 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.401669    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"]
Sep 11 16:27:13.403599 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:13.403572    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ffc983_349c_4879_883c_a7fa79fa0d12.slice/crio-25713e6de83da288bf902cb50cc08b6af315d346e226b81b2340db69d095c3bc WatchSource:0}: Error finding container 25713e6de83da288bf902cb50cc08b6af315d346e226b81b2340db69d095c3bc: Status 404 returned error can't find the container with id 25713e6de83da288bf902cb50cc08b6af315d346e226b81b2340db69d095c3bc
Sep 11 16:27:13.704559 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.704464    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerID="44932cd53fe492053fc0cf3ec51288d1181fd01174fcb2f70aa0f7988939c819" exitCode=0
Sep 11 16:27:13.704962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.704568    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw" event={"ID":"bf0baa69-0dc2-420f-be99-d18d8561fd26","Type":"ContainerDied","Data":"44932cd53fe492053fc0cf3ec51288d1181fd01174fcb2f70aa0f7988939c819"}
Sep 11 16:27:13.704962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.704615    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw" event={"ID":"bf0baa69-0dc2-420f-be99-d18d8561fd26","Type":"ContainerStarted","Data":"1c784a6121be1e7f93b3ceb10d9a529021035a39613b0ba8ca876c022a0467e9"}
Sep 11 16:27:13.706257 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.706232    2550 generic.go:356] "Generic (PLEG): container finished" podID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerID="c67d50ac02b2aa26784b34b48c12ed4ee8eede68b2e4bd70a266dc3699098017" exitCode=0
Sep 11 16:27:13.706484 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.706317    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv" event={"ID":"250bfaf5-a1f2-4ca3-8f49-480eaa824c27","Type":"ContainerDied","Data":"c67d50ac02b2aa26784b34b48c12ed4ee8eede68b2e4bd70a266dc3699098017"}
Sep 11 16:27:13.706484 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.706364    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv" event={"ID":"250bfaf5-a1f2-4ca3-8f49-480eaa824c27","Type":"ContainerStarted","Data":"802d30449c01617fe2d7d9feffc58e6ef915a5237bd51f58f52511b62464e772"}
Sep 11 16:27:13.708142 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.708047    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerID="1f0a2dcdb82f724540bf07e525e2d3447d3388c8959b15abc6722c24fd71e846" exitCode=0
Sep 11 16:27:13.708142 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.708126    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt" event={"ID":"bf146af1-c5e8-40e9-8e09-7fcb14980ebe","Type":"ContainerDied","Data":"1f0a2dcdb82f724540bf07e525e2d3447d3388c8959b15abc6722c24fd71e846"}
Sep 11 16:27:13.708259 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.708173    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt" event={"ID":"bf146af1-c5e8-40e9-8e09-7fcb14980ebe","Type":"ContainerStarted","Data":"0f504185fe4ff6f912fa4d66f9444b923b5d3843f9a37e223ea6989b9120cb7e"}
Sep 11 16:27:13.709924 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.709908    2550 generic.go:356] "Generic (PLEG): container finished" podID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerID="6aaa93e4d44f3a738f39ac1b3a64ba6a9930f3f89dd91ebb292772c5459cfcf7" exitCode=0
Sep 11 16:27:13.709998 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.709932    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp" event={"ID":"a9ffc983-349c-4879-883c-a7fa79fa0d12","Type":"ContainerDied","Data":"6aaa93e4d44f3a738f39ac1b3a64ba6a9930f3f89dd91ebb292772c5459cfcf7"}
Sep 11 16:27:13.709998 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:13.709954    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp" event={"ID":"a9ffc983-349c-4879-883c-a7fa79fa0d12","Type":"ContainerStarted","Data":"25713e6de83da288bf902cb50cc08b6af315d346e226b81b2340db69d095c3bc"}
Sep 11 16:27:14.716340 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:14.716314    2550 generic.go:356] "Generic (PLEG): container finished" podID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerID="dcb96047ec98ea4738da9ee7c212046a40fa60af511bdd621f15d2b21de70600" exitCode=0
Sep 11 16:27:14.716718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:14.716399    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv" event={"ID":"250bfaf5-a1f2-4ca3-8f49-480eaa824c27","Type":"ContainerDied","Data":"dcb96047ec98ea4738da9ee7c212046a40fa60af511bdd621f15d2b21de70600"}
Sep 11 16:27:14.718469 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:14.718444    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerID="4f5fb137a0d1f9fbe1521460b15dbeced69385f2ab29db53ca94bdb35c263309" exitCode=0
Sep 11 16:27:14.718569 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:14.718480    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt" event={"ID":"bf146af1-c5e8-40e9-8e09-7fcb14980ebe","Type":"ContainerDied","Data":"4f5fb137a0d1f9fbe1521460b15dbeced69385f2ab29db53ca94bdb35c263309"}
Sep 11 16:27:15.723988 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.723954    2550 generic.go:356] "Generic (PLEG): container finished" podID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerID="71aa886aa060bef4dfab97b84308f5725b0b031be65edd61ad6b302e64a3e775" exitCode=0
Sep 11 16:27:15.724436 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.724044    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp" event={"ID":"a9ffc983-349c-4879-883c-a7fa79fa0d12","Type":"ContainerDied","Data":"71aa886aa060bef4dfab97b84308f5725b0b031be65edd61ad6b302e64a3e775"}
Sep 11 16:27:15.726000 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.725972    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerID="41dca7581d6774d8c15431fbfaab62d07ed93fa608a7eea2a600d78d19149224" exitCode=0
Sep 11 16:27:15.726137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.726050    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw" event={"ID":"bf0baa69-0dc2-420f-be99-d18d8561fd26","Type":"ContainerDied","Data":"41dca7581d6774d8c15431fbfaab62d07ed93fa608a7eea2a600d78d19149224"}
Sep 11 16:27:15.728392 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.728370    2550 generic.go:356] "Generic (PLEG): container finished" podID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerID="cd7aad4a763616652db123b533a886ff756694b7b3f39ea7d27315e3d9607176" exitCode=0
Sep 11 16:27:15.728485 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.728450    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv" event={"ID":"250bfaf5-a1f2-4ca3-8f49-480eaa824c27","Type":"ContainerDied","Data":"cd7aad4a763616652db123b533a886ff756694b7b3f39ea7d27315e3d9607176"}
Sep 11 16:27:15.730820 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.730799    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerID="01a227dfab249681eea9e9e8280ffe17e08c84b5b2693927ad5bdb4a35fb0e68" exitCode=0
Sep 11 16:27:15.730917 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:15.730851    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt" event={"ID":"bf146af1-c5e8-40e9-8e09-7fcb14980ebe","Type":"ContainerDied","Data":"01a227dfab249681eea9e9e8280ffe17e08c84b5b2693927ad5bdb4a35fb0e68"}
Sep 11 16:27:16.736689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.736657    2550 generic.go:356] "Generic (PLEG): container finished" podID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerID="b2335c27d15ca318316167c1826302381a7e5b258167875dc95ebc191f22d88f" exitCode=0
Sep 11 16:27:16.737057 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.736739    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp" event={"ID":"a9ffc983-349c-4879-883c-a7fa79fa0d12","Type":"ContainerDied","Data":"b2335c27d15ca318316167c1826302381a7e5b258167875dc95ebc191f22d88f"}
Sep 11 16:27:16.738815 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.738794    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerID="e288413cd53c923bc710136fcb8a03b709cab170a6c39855bf625737546fe737" exitCode=0
Sep 11 16:27:16.738922 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.738879    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw" event={"ID":"bf0baa69-0dc2-420f-be99-d18d8561fd26","Type":"ContainerDied","Data":"e288413cd53c923bc710136fcb8a03b709cab170a6c39855bf625737546fe737"}
Sep 11 16:27:16.855156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.855126    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:16.885127 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.885106    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:16.982148 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982118    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtjkm\" (UniqueName: \"kubernetes.io/projected/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-kube-api-access-wtjkm\") pod \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") "
Sep 11 16:27:16.982253 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982179    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-util\") pod \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") "
Sep 11 16:27:16.982253 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982204    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-util\") pod \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") "
Sep 11 16:27:16.982345 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982258    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpxwc\" (UniqueName: \"kubernetes.io/projected/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-kube-api-access-cpxwc\") pod \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") "
Sep 11 16:27:16.982345 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982310    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-bundle\") pod \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\" (UID: \"bf146af1-c5e8-40e9-8e09-7fcb14980ebe\") "
Sep 11 16:27:16.982345 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982338    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-bundle\") pod \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\" (UID: \"250bfaf5-a1f2-4ca3-8f49-480eaa824c27\") "
Sep 11 16:27:16.982920 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.982863    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-bundle" (OuterVolumeSpecName: "bundle") pod "bf146af1-c5e8-40e9-8e09-7fcb14980ebe" (UID: "bf146af1-c5e8-40e9-8e09-7fcb14980ebe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:16.983158 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.983137    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-bundle" (OuterVolumeSpecName: "bundle") pod "250bfaf5-a1f2-4ca3-8f49-480eaa824c27" (UID: "250bfaf5-a1f2-4ca3-8f49-480eaa824c27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:16.984800 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.984780    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-kube-api-access-wtjkm" (OuterVolumeSpecName: "kube-api-access-wtjkm") pod "250bfaf5-a1f2-4ca3-8f49-480eaa824c27" (UID: "250bfaf5-a1f2-4ca3-8f49-480eaa824c27"). InnerVolumeSpecName "kube-api-access-wtjkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:16.984800 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.984792    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-kube-api-access-cpxwc" (OuterVolumeSpecName: "kube-api-access-cpxwc") pod "bf146af1-c5e8-40e9-8e09-7fcb14980ebe" (UID: "bf146af1-c5e8-40e9-8e09-7fcb14980ebe"). InnerVolumeSpecName "kube-api-access-cpxwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:16.987397 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.987336    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-util" (OuterVolumeSpecName: "util") pod "250bfaf5-a1f2-4ca3-8f49-480eaa824c27" (UID: "250bfaf5-a1f2-4ca3-8f49-480eaa824c27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:16.990592 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:16.990563    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-util" (OuterVolumeSpecName: "util") pod "bf146af1-c5e8-40e9-8e09-7fcb14980ebe" (UID: "bf146af1-c5e8-40e9-8e09-7fcb14980ebe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:17.083265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.083233    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:17.083265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.083264    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:17.083265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.083274    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxwc\" (UniqueName: \"kubernetes.io/projected/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-kube-api-access-cpxwc\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:17.083419 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.083284    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf146af1-c5e8-40e9-8e09-7fcb14980ebe-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:17.083419 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.083292    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:17.083419 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.083301    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjkm\" (UniqueName: \"kubernetes.io/projected/250bfaf5-a1f2-4ca3-8f49-480eaa824c27-kube-api-access-wtjkm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:17.745145 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.745068    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv"
Sep 11 16:27:17.745585 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.745073    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awsgkv" event={"ID":"250bfaf5-a1f2-4ca3-8f49-480eaa824c27","Type":"ContainerDied","Data":"802d30449c01617fe2d7d9feffc58e6ef915a5237bd51f58f52511b62464e772"}
Sep 11 16:27:17.745585 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.745193    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="802d30449c01617fe2d7d9feffc58e6ef915a5237bd51f58f52511b62464e772"
Sep 11 16:27:17.747100 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.747079    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt"
Sep 11 16:27:17.747215 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.747171    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c654zdt" event={"ID":"bf146af1-c5e8-40e9-8e09-7fcb14980ebe","Type":"ContainerDied","Data":"0f504185fe4ff6f912fa4d66f9444b923b5d3843f9a37e223ea6989b9120cb7e"}
Sep 11 16:27:17.747215 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.747209    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f504185fe4ff6f912fa4d66f9444b923b5d3843f9a37e223ea6989b9120cb7e"
Sep 11 16:27:17.891858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.891834    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:17.893870 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.893855    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:17.990502 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.990469    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-util\") pod \"a9ffc983-349c-4879-883c-a7fa79fa0d12\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") "
Sep 11 16:27:17.990684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.990525    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k82bw\" (UniqueName: \"kubernetes.io/projected/a9ffc983-349c-4879-883c-a7fa79fa0d12-kube-api-access-k82bw\") pod \"a9ffc983-349c-4879-883c-a7fa79fa0d12\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") "
Sep 11 16:27:17.990684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.990572    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btkt8\" (UniqueName: \"kubernetes.io/projected/bf0baa69-0dc2-420f-be99-d18d8561fd26-kube-api-access-btkt8\") pod \"bf0baa69-0dc2-420f-be99-d18d8561fd26\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") "
Sep 11 16:27:17.990684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.990590    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-util\") pod \"bf0baa69-0dc2-420f-be99-d18d8561fd26\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") "
Sep 11 16:27:17.990684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.990614    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-bundle\") pod \"a9ffc983-349c-4879-883c-a7fa79fa0d12\" (UID: \"a9ffc983-349c-4879-883c-a7fa79fa0d12\") "
Sep 11 16:27:17.990684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.990673    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-bundle\") pod \"bf0baa69-0dc2-420f-be99-d18d8561fd26\" (UID: \"bf0baa69-0dc2-420f-be99-d18d8561fd26\") "
Sep 11 16:27:17.991206 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.991175    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-bundle" (OuterVolumeSpecName: "bundle") pod "a9ffc983-349c-4879-883c-a7fa79fa0d12" (UID: "a9ffc983-349c-4879-883c-a7fa79fa0d12"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:17.991441 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.991252    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-bundle" (OuterVolumeSpecName: "bundle") pod "bf0baa69-0dc2-420f-be99-d18d8561fd26" (UID: "bf0baa69-0dc2-420f-be99-d18d8561fd26"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:17.992838 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.992809    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ffc983-349c-4879-883c-a7fa79fa0d12-kube-api-access-k82bw" (OuterVolumeSpecName: "kube-api-access-k82bw") pod "a9ffc983-349c-4879-883c-a7fa79fa0d12" (UID: "a9ffc983-349c-4879-883c-a7fa79fa0d12"). InnerVolumeSpecName "kube-api-access-k82bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:17.992904 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.992891    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0baa69-0dc2-420f-be99-d18d8561fd26-kube-api-access-btkt8" (OuterVolumeSpecName: "kube-api-access-btkt8") pod "bf0baa69-0dc2-420f-be99-d18d8561fd26" (UID: "bf0baa69-0dc2-420f-be99-d18d8561fd26"). InnerVolumeSpecName "kube-api-access-btkt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:17.996528 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.996474    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-util" (OuterVolumeSpecName: "util") pod "a9ffc983-349c-4879-883c-a7fa79fa0d12" (UID: "a9ffc983-349c-4879-883c-a7fa79fa0d12"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:17.998953 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:17.998931    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-util" (OuterVolumeSpecName: "util") pod "bf0baa69-0dc2-420f-be99-d18d8561fd26" (UID: "bf0baa69-0dc2-420f-be99-d18d8561fd26"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:27:18.091620 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.091600    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:18.091620 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.091619    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:18.091725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.091628    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k82bw\" (UniqueName: \"kubernetes.io/projected/a9ffc983-349c-4879-883c-a7fa79fa0d12-kube-api-access-k82bw\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:18.091725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.091637    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkt8\" (UniqueName: \"kubernetes.io/projected/bf0baa69-0dc2-420f-be99-d18d8561fd26-kube-api-access-btkt8\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:18.091725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.091645    2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf0baa69-0dc2-420f-be99-d18d8561fd26-util\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:18.091725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.091653    2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9ffc983-349c-4879-883c-a7fa79fa0d12-bundle\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:27:18.753754 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.753726    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp"
Sep 11 16:27:18.754226 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.753750    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gccp" event={"ID":"a9ffc983-349c-4879-883c-a7fa79fa0d12","Type":"ContainerDied","Data":"25713e6de83da288bf902cb50cc08b6af315d346e226b81b2340db69d095c3bc"}
Sep 11 16:27:18.754226 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.753804    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25713e6de83da288bf902cb50cc08b6af315d346e226b81b2340db69d095c3bc"
Sep 11 16:27:18.755760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.755738    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw" event={"ID":"bf0baa69-0dc2-420f-be99-d18d8561fd26","Type":"ContainerDied","Data":"1c784a6121be1e7f93b3ceb10d9a529021035a39613b0ba8ca876c022a0467e9"}
Sep 11 16:27:18.755868 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.755772    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c784a6121be1e7f93b3ceb10d9a529021035a39613b0ba8ca876c022a0467e9"
Sep 11 16:27:18.755868 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:18.755775    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cfr9rw"
Sep 11 16:27:25.180078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180036    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"]
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180587    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180608    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180628    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180636    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180647    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="pull"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180655    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="pull"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180671    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180679    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180687    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180696    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180716    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180724    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180732    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180740    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180754    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180768    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="util"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180778    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="pull"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180785    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="pull"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180793    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180801    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="extract"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180819    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="pull"
Sep 11 16:27:25.180842 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180827    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="pull"
Sep 11 16:27:25.181846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180863    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="pull"
Sep 11 16:27:25.181846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.180871    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="pull"
Sep 11 16:27:25.181846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.181196    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ffc983-349c-4879-883c-a7fa79fa0d12" containerName="extract"
Sep 11 16:27:25.181846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.181222    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="250bfaf5-a1f2-4ca3-8f49-480eaa824c27" containerName="extract"
Sep 11 16:27:25.181846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.181232    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf146af1-c5e8-40e9-8e09-7fcb14980ebe" containerName="extract"
Sep 11 16:27:25.181846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.181315    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0baa69-0dc2-420f-be99-d18d8561fd26" containerName="extract"
Sep 11 16:27:25.185623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.185603    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:25.188016 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.187996    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 16:27:25.188855 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.188835    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9qvzl\""
Sep 11 16:27:25.188936 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.188834    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 16:27:25.194804 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.194781    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"]
Sep 11 16:27:25.343170 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.343139    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5gh\" (UniqueName: \"kubernetes.io/projected/ee02f132-e49c-41cb-bf07-22d37a07bd03-kube-api-access-5k5gh\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qcd7n\" (UID: \"ee02f132-e49c-41cb-bf07-22d37a07bd03\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:25.443924 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.443851    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5gh\" (UniqueName: \"kubernetes.io/projected/ee02f132-e49c-41cb-bf07-22d37a07bd03-kube-api-access-5k5gh\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qcd7n\" (UID: \"ee02f132-e49c-41cb-bf07-22d37a07bd03\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:25.453450 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.453425    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5gh\" (UniqueName: \"kubernetes.io/projected/ee02f132-e49c-41cb-bf07-22d37a07bd03-kube-api-access-5k5gh\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-qcd7n\" (UID: \"ee02f132-e49c-41cb-bf07-22d37a07bd03\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:25.495676 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.495641    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:25.767223 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.767198    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"]
Sep 11 16:27:25.768754 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:25.768721    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee02f132_e49c_41cb_bf07_22d37a07bd03.slice/crio-abd21c9c4eda1efc37610567630e8fd8131fedb4a76cc002ee43844b5af9c9b6 WatchSource:0}: Error finding container abd21c9c4eda1efc37610567630e8fd8131fedb4a76cc002ee43844b5af9c9b6: Status 404 returned error can't find the container with id abd21c9c4eda1efc37610567630e8fd8131fedb4a76cc002ee43844b5af9c9b6
Sep 11 16:27:25.785880 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:25.785857    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n" event={"ID":"ee02f132-e49c-41cb-bf07-22d37a07bd03","Type":"ContainerStarted","Data":"abd21c9c4eda1efc37610567630e8fd8131fedb4a76cc002ee43844b5af9c9b6"}
Sep 11 16:27:27.489207 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.489182    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"]
Sep 11 16:27:27.493439 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.493421    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:27.495200 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.495184    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 16:27:27.495273 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.495239    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wlkzc\""
Sep 11 16:27:27.501960 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.501942    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"]
Sep 11 16:27:27.660875 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.660838    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46x2\" (UniqueName: \"kubernetes.io/projected/3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb-kube-api-access-r46x2\") pod \"dns-operator-controller-manager-65dc8f76f5-sdrps\" (UID: \"3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:27.761858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.761774    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46x2\" (UniqueName: \"kubernetes.io/projected/3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb-kube-api-access-r46x2\") pod \"dns-operator-controller-manager-65dc8f76f5-sdrps\" (UID: \"3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:27.769246 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.769215    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46x2\" (UniqueName: \"kubernetes.io/projected/3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb-kube-api-access-r46x2\") pod \"dns-operator-controller-manager-65dc8f76f5-sdrps\" (UID: \"3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:27.797137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.797105    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n" event={"ID":"ee02f132-e49c-41cb-bf07-22d37a07bd03","Type":"ContainerStarted","Data":"fd3bd626f10f4369ca5694aead0b2a3046a69469ef656fd72da81a52f71bb7d3"}
Sep 11 16:27:27.797236 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.797177    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:27.809485 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.809461    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:27.819469 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:27.819428    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n" podStartSLOduration=1.138403063 podStartE2EDuration="2.819415238s" totalImagesPullingTime="1.681012175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:27.818464379 +0000 UTC m=+622.774512008" watchObservedRunningTime="2026-09-11 16:27:27.819415238 +0000 UTC m=+622.775462866"
Sep 11 16:27:28.080933 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:28.080911    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"]
Sep 11 16:27:28.083174 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:28.083147    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca5357c_cbeb_4ea6_b72d_3163db0d2cdb.slice/crio-5cae273afd0762d35fcc83e0d34e7bfa2346eb1ec63cf062aafc4cc16b727820 WatchSource:0}: Error finding container 5cae273afd0762d35fcc83e0d34e7bfa2346eb1ec63cf062aafc4cc16b727820: Status 404 returned error can't find the container with id 5cae273afd0762d35fcc83e0d34e7bfa2346eb1ec63cf062aafc4cc16b727820
Sep 11 16:27:28.803265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:28.803233    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps" event={"ID":"3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb","Type":"ContainerStarted","Data":"5cae273afd0762d35fcc83e0d34e7bfa2346eb1ec63cf062aafc4cc16b727820"}
Sep 11 16:27:29.808915 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:29.808879    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps" event={"ID":"3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb","Type":"ContainerStarted","Data":"1458c37f020d98964a0c03742fe8d58bef3db6cecfa28d23990e1026dcd9e0ba"}
Sep 11 16:27:29.809360 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:29.808999    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:29.831368 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:29.831324    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps" podStartSLOduration=1.276072058 podStartE2EDuration="2.831312024s" totalImagesPullingTime="1.555239966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:29.829019618 +0000 UTC m=+624.785067241" watchObservedRunningTime="2026-09-11 16:27:29.831312024 +0000 UTC m=+624.787359652"
Sep 11 16:27:31.290114 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.290070    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-64hpq"]
Sep 11 16:27:31.293537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.293512    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:27:31.295573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.295553    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xd89j\""
Sep 11 16:27:31.309724 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.309702    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-64hpq"]
Sep 11 16:27:31.488174 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.488144    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2fl\" (UniqueName: \"kubernetes.io/projected/3986030a-ec5f-4a4c-85bc-e49ffd8e2581-kube-api-access-ff2fl\") pod \"authorino-operator-6fc8f864f4-64hpq\" (UID: \"3986030a-ec5f-4a4c-85bc-e49ffd8e2581\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:27:31.589422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.589404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2fl\" (UniqueName: \"kubernetes.io/projected/3986030a-ec5f-4a4c-85bc-e49ffd8e2581-kube-api-access-ff2fl\") pod \"authorino-operator-6fc8f864f4-64hpq\" (UID: \"3986030a-ec5f-4a4c-85bc-e49ffd8e2581\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:27:31.600058 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.600036    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2fl\" (UniqueName: \"kubernetes.io/projected/3986030a-ec5f-4a4c-85bc-e49ffd8e2581-kube-api-access-ff2fl\") pod \"authorino-operator-6fc8f864f4-64hpq\" (UID: \"3986030a-ec5f-4a4c-85bc-e49ffd8e2581\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:27:31.603821 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.603801    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:27:31.879758 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:31.879735    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-64hpq"]
Sep 11 16:27:31.882292 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:27:31.882261    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3986030a_ec5f_4a4c_85bc_e49ffd8e2581.slice/crio-c2a1bc83a220d5b203f8d14f0fd264d253700bf64a0a585b2cb4133f03da34e4 WatchSource:0}: Error finding container c2a1bc83a220d5b203f8d14f0fd264d253700bf64a0a585b2cb4133f03da34e4: Status 404 returned error can't find the container with id c2a1bc83a220d5b203f8d14f0fd264d253700bf64a0a585b2cb4133f03da34e4
Sep 11 16:27:32.829222 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:32.829185    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq" event={"ID":"3986030a-ec5f-4a4c-85bc-e49ffd8e2581","Type":"ContainerStarted","Data":"c2a1bc83a220d5b203f8d14f0fd264d253700bf64a0a585b2cb4133f03da34e4"}
Sep 11 16:27:33.835117 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:33.835083    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq" event={"ID":"3986030a-ec5f-4a4c-85bc-e49ffd8e2581","Type":"ContainerStarted","Data":"b1b26fc0e14bbebaa98d118668b16fda4bb750947ebfd3a2ac8d9b8a62980307"}
Sep 11 16:27:33.835490 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:33.835219    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:27:33.852986 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:33.852942    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq" podStartSLOduration=1.336507848 podStartE2EDuration="2.852927097s" totalImagesPullingTime="1.516419249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:33.851516591 +0000 UTC m=+628.807564219" watchObservedRunningTime="2026-09-11 16:27:33.852927097 +0000 UTC m=+628.808974725"
Sep 11 16:27:38.805467 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:38.805435    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-qcd7n"
Sep 11 16:27:40.815173 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:40.815139    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-sdrps"
Sep 11 16:27:44.842324 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:27:44.842286    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-64hpq"
Sep 11 16:28:26.063527 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.063498    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:26.066216 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.066192    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.069952 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.069933    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 16:28:26.070072 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.070001    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2r47n\""
Sep 11 16:28:26.081474 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.081454    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:26.144984 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.144959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d69f9e7d-428f-436f-ba10-fc5d929b1763-config-file\") pod \"limitador-limitador-8588749977-v7twf\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") " pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.145125 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.144994    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl2r\" (UniqueName: \"kubernetes.io/projected/d69f9e7d-428f-436f-ba10-fc5d929b1763-kube-api-access-hjl2r\") pod \"limitador-limitador-8588749977-v7twf\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") " pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.156001 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.155975    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:26.245408 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.245380    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d69f9e7d-428f-436f-ba10-fc5d929b1763-config-file\") pod \"limitador-limitador-8588749977-v7twf\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") " pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.245532 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.245415    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl2r\" (UniqueName: \"kubernetes.io/projected/d69f9e7d-428f-436f-ba10-fc5d929b1763-kube-api-access-hjl2r\") pod \"limitador-limitador-8588749977-v7twf\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") " pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.245992 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.245974    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d69f9e7d-428f-436f-ba10-fc5d929b1763-config-file\") pod \"limitador-limitador-8588749977-v7twf\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") " pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.253038 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.253014    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl2r\" (UniqueName: \"kubernetes.io/projected/d69f9e7d-428f-436f-ba10-fc5d929b1763-kube-api-access-hjl2r\") pod \"limitador-limitador-8588749977-v7twf\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") " pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.375452 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.375425    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:26.645866 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:26.645844    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:26.647851 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:28:26.647826    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69f9e7d_428f_436f_ba10_fc5d929b1763.slice/crio-1983191297bf1a218bf1b9ab10da3ea93eb74dd74c391fb3ec46dfd27910f89c WatchSource:0}: Error finding container 1983191297bf1a218bf1b9ab10da3ea93eb74dd74c391fb3ec46dfd27910f89c: Status 404 returned error can't find the container with id 1983191297bf1a218bf1b9ab10da3ea93eb74dd74c391fb3ec46dfd27910f89c
Sep 11 16:28:27.071469 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:27.071393    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v7twf" event={"ID":"d69f9e7d-428f-436f-ba10-fc5d929b1763","Type":"ContainerStarted","Data":"1983191297bf1a218bf1b9ab10da3ea93eb74dd74c391fb3ec46dfd27910f89c"}
Sep 11 16:28:28.076835 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:28.076807    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v7twf" event={"ID":"d69f9e7d-428f-436f-ba10-fc5d929b1763","Type":"ContainerStarted","Data":"dad15fade78c1b64cfac125b317738bddc13a31e9c08d44d5ee1029022337654"}
Sep 11 16:28:28.077137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:28.076953    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:28.092789 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:28.092749    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-v7twf" podStartSLOduration=0.737492159 podStartE2EDuration="2.092711894s" totalImagesPullingTime="1.355219735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:28.091318098 +0000 UTC m=+683.047365726" watchObservedRunningTime="2026-09-11 16:28:28.092711894 +0000 UTC m=+683.048759524"
Sep 11 16:28:39.083149 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:39.083111    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:39.629466 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:39.629433    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:39.629636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:39.629608    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-v7twf" podUID="d69f9e7d-428f-436f-ba10-fc5d929b1763" containerName="limitador" containerID="cri-o://dad15fade78c1b64cfac125b317738bddc13a31e9c08d44d5ee1029022337654" gracePeriod=30
Sep 11 16:28:40.131164 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.131135    2550 generic.go:356] "Generic (PLEG): container finished" podID="d69f9e7d-428f-436f-ba10-fc5d929b1763" containerID="dad15fade78c1b64cfac125b317738bddc13a31e9c08d44d5ee1029022337654" exitCode=0
Sep 11 16:28:40.131496 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.131210    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v7twf" event={"ID":"d69f9e7d-428f-436f-ba10-fc5d929b1763","Type":"ContainerDied","Data":"dad15fade78c1b64cfac125b317738bddc13a31e9c08d44d5ee1029022337654"}
Sep 11 16:28:40.167304 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.167284    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:40.240090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.240018    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d69f9e7d-428f-436f-ba10-fc5d929b1763-config-file\") pod \"d69f9e7d-428f-436f-ba10-fc5d929b1763\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") "
Sep 11 16:28:40.240090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.240069    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjl2r\" (UniqueName: \"kubernetes.io/projected/d69f9e7d-428f-436f-ba10-fc5d929b1763-kube-api-access-hjl2r\") pod \"d69f9e7d-428f-436f-ba10-fc5d929b1763\" (UID: \"d69f9e7d-428f-436f-ba10-fc5d929b1763\") "
Sep 11 16:28:40.240359 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.240334    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69f9e7d-428f-436f-ba10-fc5d929b1763-config-file" (OuterVolumeSpecName: "config-file") pod "d69f9e7d-428f-436f-ba10-fc5d929b1763" (UID: "d69f9e7d-428f-436f-ba10-fc5d929b1763"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:28:40.242152 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.242130    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69f9e7d-428f-436f-ba10-fc5d929b1763-kube-api-access-hjl2r" (OuterVolumeSpecName: "kube-api-access-hjl2r") pod "d69f9e7d-428f-436f-ba10-fc5d929b1763" (UID: "d69f9e7d-428f-436f-ba10-fc5d929b1763"). InnerVolumeSpecName "kube-api-access-hjl2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:40.340429 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.340400    2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d69f9e7d-428f-436f-ba10-fc5d929b1763-config-file\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:28:40.340429 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:40.340424    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjl2r\" (UniqueName: \"kubernetes.io/projected/d69f9e7d-428f-436f-ba10-fc5d929b1763-kube-api-access-hjl2r\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:28:41.137229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:41.137194    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v7twf" event={"ID":"d69f9e7d-428f-436f-ba10-fc5d929b1763","Type":"ContainerDied","Data":"1983191297bf1a218bf1b9ab10da3ea93eb74dd74c391fb3ec46dfd27910f89c"}
Sep 11 16:28:41.137229 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:41.137234    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v7twf"
Sep 11 16:28:41.137756 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:41.137264    2550 scope.go:117] "RemoveContainer" containerID="dad15fade78c1b64cfac125b317738bddc13a31e9c08d44d5ee1029022337654"
Sep 11 16:28:41.156038 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:41.156014    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:41.160244 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:41.160221    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v7twf"]
Sep 11 16:28:41.477059 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:28:41.476986    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d69f9e7d-428f-436f-ba10-fc5d929b1763" path="/var/lib/kubelet/pods/d69f9e7d-428f-436f-ba10-fc5d929b1763/volumes"
Sep 11 16:29:07.528678 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.528650    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"]
Sep 11 16:29:07.529093 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.528937    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d69f9e7d-428f-436f-ba10-fc5d929b1763" containerName="limitador"
Sep 11 16:29:07.529093 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.528947    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d69f9e7d-428f-436f-ba10-fc5d929b1763" containerName="limitador"
Sep 11 16:29:07.529093 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.528997    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d69f9e7d-428f-436f-ba10-fc5d929b1763" containerName="limitador"
Sep 11 16:29:07.531371 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.531351    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.545783 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.545762    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"]
Sep 11 16:29:07.627283 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627259    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.627430 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627288    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.627430 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.627430 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627331    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfhb\" (UniqueName: \"kubernetes.io/projected/d2c4af5c-e523-46ac-a53b-93b38efccf5d-kube-api-access-pdfhb\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.627596 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627434    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.627596 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627488    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c4af5c-e523-46ac-a53b-93b38efccf5d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.627596 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.627574    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.728800 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.728903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728811    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c4af5c-e523-46ac-a53b-93b38efccf5d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.728903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728852    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.728903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728881    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.729035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728906    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.729035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728938    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.729035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.728962    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfhb\" (UniqueName: \"kubernetes.io/projected/d2c4af5c-e523-46ac-a53b-93b38efccf5d-kube-api-access-pdfhb\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.729815 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.729788    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.731282 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.731259    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.731384 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.731334    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c4af5c-e523-46ac-a53b-93b38efccf5d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.731456 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.731412    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.731513 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.731496    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.736099 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.736079    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c4af5c-e523-46ac-a53b-93b38efccf5d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.736747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.736730    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfhb\" (UniqueName: \"kubernetes.io/projected/d2c4af5c-e523-46ac-a53b-93b38efccf5d-kube-api-access-pdfhb\") pod \"istiod-openshift-gateway-5677ccb49b-8f972\" (UID: \"d2c4af5c-e523-46ac-a53b-93b38efccf5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:07.840667 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:07.840637    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:08.113272 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.113250    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"]
Sep 11 16:29:08.114780 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:29:08.114751    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c4af5c_e523_46ac_a53b_93b38efccf5d.slice/crio-54d5af9057d94a26cb50b660134ca4dd59311e575bb9fe79697fe8d11b2fb79a WatchSource:0}: Error finding container 54d5af9057d94a26cb50b660134ca4dd59311e575bb9fe79697fe8d11b2fb79a: Status 404 returned error can't find the container with id 54d5af9057d94a26cb50b660134ca4dd59311e575bb9fe79697fe8d11b2fb79a
Sep 11 16:29:08.117001 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.116969    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 16:29:08.117097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.117043    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 16:29:08.264159 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.264124    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972" event={"ID":"d2c4af5c-e523-46ac-a53b-93b38efccf5d","Type":"ContainerStarted","Data":"78d2e56a4241e6d817d46d9147cedcf1ea902cf05c615d5caff6e9e6522d429a"}
Sep 11 16:29:08.264413 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.264181    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972" event={"ID":"d2c4af5c-e523-46ac-a53b-93b38efccf5d","Type":"ContainerStarted","Data":"54d5af9057d94a26cb50b660134ca4dd59311e575bb9fe79697fe8d11b2fb79a"}
Sep 11 16:29:08.264413 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.264207    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:08.308257 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:08.308207    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972" podStartSLOduration=1.308181888 podStartE2EDuration="1.308181888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:08.305996468 +0000 UTC m=+723.262044095" watchObservedRunningTime="2026-09-11 16:29:08.308181888 +0000 UTC m=+723.264229514"
Sep 11 16:29:09.271047 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.271020    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8f972"
Sep 11 16:29:09.322156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.322128    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"]
Sep 11 16:29:09.322323 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.322304    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" podUID="99b8aa1c-ffad-420b-9f6f-a962c6519215" containerName="discovery" containerID="cri-o://fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c" gracePeriod=30
Sep 11 16:29:09.569164 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.569139    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:29:09.645077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645019    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-cacerts\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645074    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-ca-configmap\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645106    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-kubeconfig\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645137    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-kube-api-access-25sxl\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645169    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8aa1c-ffad-420b-9f6f-a962c6519215-local-certs\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645284    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-token\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645342    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-dns-cert\") pod \"99b8aa1c-ffad-420b-9f6f-a962c6519215\" (UID: \"99b8aa1c-ffad-420b-9f6f-a962c6519215\") "
Sep 11 16:29:09.645625 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.645591    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:29:09.647568 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.647498    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:09.647683 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.647563    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b8aa1c-ffad-420b-9f6f-a962c6519215-local-certs" (OuterVolumeSpecName: "local-certs") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:29:09.647683 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.647673    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-cacerts" (OuterVolumeSpecName: "cacerts") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:09.647789 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.647741    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-token" (OuterVolumeSpecName: "istio-token") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:29:09.647789 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.647766    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-kube-api-access-25sxl" (OuterVolumeSpecName: "kube-api-access-25sxl") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "kube-api-access-25sxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:29:09.647904 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.647830    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "99b8aa1c-ffad-420b-9f6f-a962c6519215" (UID: "99b8aa1c-ffad-420b-9f6f-a962c6519215"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:09.746456 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746429    2550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-cacerts\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.746456 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746453    2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-ca-configmap\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.746622 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746470    2550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-kubeconfig\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.746622 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746482    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-kube-api-access-25sxl\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.746622 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746495    2550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b8aa1c-ffad-420b-9f6f-a962c6519215-local-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.746622 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746506    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-token\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.746622 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:09.746517    2550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99b8aa1c-ffad-420b-9f6f-a962c6519215-istio-csr-dns-cert\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:29:10.274410 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.274366    2550 generic.go:356] "Generic (PLEG): container finished" podID="99b8aa1c-ffad-420b-9f6f-a962c6519215" containerID="fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c" exitCode=0
Sep 11 16:29:10.274828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.274429    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"
Sep 11 16:29:10.274828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.274450    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" event={"ID":"99b8aa1c-ffad-420b-9f6f-a962c6519215","Type":"ContainerDied","Data":"fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c"}
Sep 11 16:29:10.274828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.274560    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf" event={"ID":"99b8aa1c-ffad-420b-9f6f-a962c6519215","Type":"ContainerDied","Data":"200cd2e0f2f4a964ba1fd60f34679fe20c2911fb0c64f6106cc1302cb71d76bb"}
Sep 11 16:29:10.274828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.274598    2550 scope.go:117] "RemoveContainer" containerID="fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c"
Sep 11 16:29:10.283812 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.283793    2550 scope.go:117] "RemoveContainer" containerID="fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c"
Sep 11 16:29:10.284082 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:29:10.284064    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c\": container with ID starting with fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c not found: ID does not exist" containerID="fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c"
Sep 11 16:29:10.284117 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.284090    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c"} err="failed to get container status \"fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c\": rpc error: code = NotFound desc = could not find container \"fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c\": container with ID starting with fe21e4907514c9eaebb9896a301d845030caec5c587f4d43d0d49ba28852607c not found: ID does not exist"
Sep 11 16:29:10.295742 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.295719    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"]
Sep 11 16:29:10.299477 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:10.299453    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xcfxf"]
Sep 11 16:29:11.476017 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:11.475984    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b8aa1c-ffad-420b-9f6f-a962c6519215" path="/var/lib/kubelet/pods/99b8aa1c-ffad-420b-9f6f-a962c6519215/volumes"
Sep 11 16:29:19.104417 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.104386    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-bqcrn"]
Sep 11 16:29:19.104771 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.104683    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b8aa1c-ffad-420b-9f6f-a962c6519215" containerName="discovery"
Sep 11 16:29:19.104771 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.104694    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b8aa1c-ffad-420b-9f6f-a962c6519215" containerName="discovery"
Sep 11 16:29:19.104771 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.104749    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b8aa1c-ffad-420b-9f6f-a962c6519215" containerName="discovery"
Sep 11 16:29:19.107137 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.107119    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.109209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.109187    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:29:19.109332 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.109256    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:29:19.109604 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.109580    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qnn9t\""
Sep 11 16:29:19.109604 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.109603    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 11 16:29:19.116128 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.116106    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-bqcrn"]
Sep 11 16:29:19.153389 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.153368    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-lghrk"]
Sep 11 16:29:19.155509 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.155492    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.157216 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.157198    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 11 16:29:19.157311 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.157250    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-k9t7c\""
Sep 11 16:29:19.164488 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.164468    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lghrk"]
Sep 11 16:29:19.206912 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.206887    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.207010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.206920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z55z2\" (UniqueName: \"kubernetes.io/projected/714e8623-bddb-4ec8-867a-9002952a8058-kube-api-access-z55z2\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.308167 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.308141    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.308259 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.308173    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z55z2\" (UniqueName: \"kubernetes.io/projected/714e8623-bddb-4ec8-867a-9002952a8058-kube-api-access-z55z2\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.308259 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.308205    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b20cbee5-2792-4ed2-a389-5e482ad054fb-data\") pod \"seaweedfs-64568bcd49-lghrk\" (UID: \"b20cbee5-2792-4ed2-a389-5e482ad054fb\") " pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.308259 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.308231    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbdh\" (UniqueName: \"kubernetes.io/projected/b20cbee5-2792-4ed2-a389-5e482ad054fb-kube-api-access-fkbdh\") pod \"seaweedfs-64568bcd49-lghrk\" (UID: \"b20cbee5-2792-4ed2-a389-5e482ad054fb\") " pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.308364 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:29:19.308287    2550 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 16:29:19.308364 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:29:19.308349    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert podName:714e8623-bddb-4ec8-867a-9002952a8058 nodeName:}" failed. No retries permitted until 2026-09-11 16:29:19.808332391 +0000 UTC m=+734.764380001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert") pod "kserve-controller-manager-656b75fc8d-bqcrn" (UID: "714e8623-bddb-4ec8-867a-9002952a8058") : secret "kserve-webhook-server-cert" not found
Sep 11 16:29:19.316156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.316133    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z55z2\" (UniqueName: \"kubernetes.io/projected/714e8623-bddb-4ec8-867a-9002952a8058-kube-api-access-z55z2\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.408593 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.408525    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b20cbee5-2792-4ed2-a389-5e482ad054fb-data\") pod \"seaweedfs-64568bcd49-lghrk\" (UID: \"b20cbee5-2792-4ed2-a389-5e482ad054fb\") " pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.408593 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.408581    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbdh\" (UniqueName: \"kubernetes.io/projected/b20cbee5-2792-4ed2-a389-5e482ad054fb-kube-api-access-fkbdh\") pod \"seaweedfs-64568bcd49-lghrk\" (UID: \"b20cbee5-2792-4ed2-a389-5e482ad054fb\") " pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.408852 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.408833    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b20cbee5-2792-4ed2-a389-5e482ad054fb-data\") pod \"seaweedfs-64568bcd49-lghrk\" (UID: \"b20cbee5-2792-4ed2-a389-5e482ad054fb\") " pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.421909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.421879    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbdh\" (UniqueName: \"kubernetes.io/projected/b20cbee5-2792-4ed2-a389-5e482ad054fb-kube-api-access-fkbdh\") pod \"seaweedfs-64568bcd49-lghrk\" (UID: \"b20cbee5-2792-4ed2-a389-5e482ad054fb\") " pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.464346 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.464321    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:19.739218 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.739141    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lghrk"]
Sep 11 16:29:19.742006 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:29:19.741980    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20cbee5_2792_4ed2_a389_5e482ad054fb.slice/crio-d2bd0ab85e4744a339aa7940e7eb55b678c68b17d9c71ca38380281cf473f030 WatchSource:0}: Error finding container d2bd0ab85e4744a339aa7940e7eb55b678c68b17d9c71ca38380281cf473f030: Status 404 returned error can't find the container with id d2bd0ab85e4744a339aa7940e7eb55b678c68b17d9c71ca38380281cf473f030
Sep 11 16:29:19.811701 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.811672    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:19.813910 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:19.813886    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert\") pod \"kserve-controller-manager-656b75fc8d-bqcrn\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") " pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:20.017917 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:20.017856    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:20.284819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:20.284796    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-bqcrn"]
Sep 11 16:29:20.286477 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:29:20.286446    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714e8623_bddb_4ec8_867a_9002952a8058.slice/crio-6635f1ae17374a046ee27fdf6b9a8571eddb38105e254345603ccb2d79d1d2b3 WatchSource:0}: Error finding container 6635f1ae17374a046ee27fdf6b9a8571eddb38105e254345603ccb2d79d1d2b3: Status 404 returned error can't find the container with id 6635f1ae17374a046ee27fdf6b9a8571eddb38105e254345603ccb2d79d1d2b3
Sep 11 16:29:20.323664 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:20.323621    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn" event={"ID":"714e8623-bddb-4ec8-867a-9002952a8058","Type":"ContainerStarted","Data":"6635f1ae17374a046ee27fdf6b9a8571eddb38105e254345603ccb2d79d1d2b3"}
Sep 11 16:29:20.325110 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:20.325074    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lghrk" event={"ID":"b20cbee5-2792-4ed2-a389-5e482ad054fb","Type":"ContainerStarted","Data":"d2bd0ab85e4744a339aa7940e7eb55b678c68b17d9c71ca38380281cf473f030"}
Sep 11 16:29:24.354300 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:24.354266    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn" event={"ID":"714e8623-bddb-4ec8-867a-9002952a8058","Type":"ContainerStarted","Data":"6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8"}
Sep 11 16:29:24.354753 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:24.354397    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:29:24.356125 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:24.356092    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lghrk" event={"ID":"b20cbee5-2792-4ed2-a389-5e482ad054fb","Type":"ContainerStarted","Data":"46247d14e6c2a7d12e620bab5dca7762aa279828a66a3aea2949f2c440894d61"}
Sep 11 16:29:24.356261 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:24.356136    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:24.368687 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:24.368644    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn" podStartSLOduration=1.8248655980000001 podStartE2EDuration="5.36862915s" totalImagesPullingTime="3.543763552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:24.368423537 +0000 UTC m=+739.324471166" watchObservedRunningTime="2026-09-11 16:29:24.36862915 +0000 UTC m=+739.324676779"
Sep 11 16:29:24.382784 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:24.382749    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-lghrk" podStartSLOduration=1.244065159 podStartE2EDuration="5.38273661s" totalImagesPullingTime="4.138671451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:24.381484658 +0000 UTC m=+739.337532286" watchObservedRunningTime="2026-09-11 16:29:24.38273661 +0000 UTC m=+739.338784237"
Sep 11 16:29:30.363036 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:30.363005    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-lghrk"
Sep 11 16:29:55.365260 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:29:55.365226    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:30:02.542645 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.542613    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-bqcrn"]
Sep 11 16:30:02.543100 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.542806    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn" podUID="714e8623-bddb-4ec8-867a-9002952a8058" containerName="manager" containerID="cri-o://6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8" gracePeriod=10
Sep 11 16:30:02.564844 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.564818    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-p8vmj"]
Sep 11 16:30:02.566875 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.566854    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.577931 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.577909    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-p8vmj"]
Sep 11 16:30:02.614015 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.613993    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b471abb-626e-4430-b15c-57694b24b497-cert\") pod \"kserve-controller-manager-656b75fc8d-p8vmj\" (UID: \"9b471abb-626e-4430-b15c-57694b24b497\") " pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.614120 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.614027    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsgdp\" (UniqueName: \"kubernetes.io/projected/9b471abb-626e-4430-b15c-57694b24b497-kube-api-access-bsgdp\") pod \"kserve-controller-manager-656b75fc8d-p8vmj\" (UID: \"9b471abb-626e-4430-b15c-57694b24b497\") " pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.714869 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.714837    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b471abb-626e-4430-b15c-57694b24b497-cert\") pod \"kserve-controller-manager-656b75fc8d-p8vmj\" (UID: \"9b471abb-626e-4430-b15c-57694b24b497\") " pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.715012 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.714903    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgdp\" (UniqueName: \"kubernetes.io/projected/9b471abb-626e-4430-b15c-57694b24b497-kube-api-access-bsgdp\") pod \"kserve-controller-manager-656b75fc8d-p8vmj\" (UID: \"9b471abb-626e-4430-b15c-57694b24b497\") " pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.717709 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.717684    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b471abb-626e-4430-b15c-57694b24b497-cert\") pod \"kserve-controller-manager-656b75fc8d-p8vmj\" (UID: \"9b471abb-626e-4430-b15c-57694b24b497\") " pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.722075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.722051    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgdp\" (UniqueName: \"kubernetes.io/projected/9b471abb-626e-4430-b15c-57694b24b497-kube-api-access-bsgdp\") pod \"kserve-controller-manager-656b75fc8d-p8vmj\" (UID: \"9b471abb-626e-4430-b15c-57694b24b497\") " pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.780201 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.780180    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:30:02.815370 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.815308    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert\") pod \"714e8623-bddb-4ec8-867a-9002952a8058\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") "
Sep 11 16:30:02.815370 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.815360    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z55z2\" (UniqueName: \"kubernetes.io/projected/714e8623-bddb-4ec8-867a-9002952a8058-kube-api-access-z55z2\") pod \"714e8623-bddb-4ec8-867a-9002952a8058\" (UID: \"714e8623-bddb-4ec8-867a-9002952a8058\") "
Sep 11 16:30:02.817609 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.817574    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714e8623-bddb-4ec8-867a-9002952a8058-kube-api-access-z55z2" (OuterVolumeSpecName: "kube-api-access-z55z2") pod "714e8623-bddb-4ec8-867a-9002952a8058" (UID: "714e8623-bddb-4ec8-867a-9002952a8058"). InnerVolumeSpecName "kube-api-access-z55z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:30:02.817759 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.817734    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert" (OuterVolumeSpecName: "cert") pod "714e8623-bddb-4ec8-867a-9002952a8058" (UID: "714e8623-bddb-4ec8-867a-9002952a8058"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:30:02.912844 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.912815    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:02.916807 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.916771    2550 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8623-bddb-4ec8-867a-9002952a8058-cert\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:30:02.916807 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:02.916803    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z55z2\" (UniqueName: \"kubernetes.io/projected/714e8623-bddb-4ec8-867a-9002952a8058-kube-api-access-z55z2\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:30:03.179125 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.179097    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-p8vmj"]
Sep 11 16:30:03.182303 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:30:03.182275    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b471abb_626e_4430_b15c_57694b24b497.slice/crio-fecac397d96d6fe4bd7cda0d6a877221cd4d915cf905d3ef0f6b1ebef89f9a3e WatchSource:0}: Error finding container fecac397d96d6fe4bd7cda0d6a877221cd4d915cf905d3ef0f6b1ebef89f9a3e: Status 404 returned error can't find the container with id fecac397d96d6fe4bd7cda0d6a877221cd4d915cf905d3ef0f6b1ebef89f9a3e
Sep 11 16:30:03.183643 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.183616    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:30:03.545314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.545272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj" event={"ID":"9b471abb-626e-4430-b15c-57694b24b497","Type":"ContainerStarted","Data":"fecac397d96d6fe4bd7cda0d6a877221cd4d915cf905d3ef0f6b1ebef89f9a3e"}
Sep 11 16:30:03.546594 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.546570    2550 generic.go:356] "Generic (PLEG): container finished" podID="714e8623-bddb-4ec8-867a-9002952a8058" containerID="6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8" exitCode=0
Sep 11 16:30:03.546702 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.546620    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn"
Sep 11 16:30:03.546702 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.546655    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn" event={"ID":"714e8623-bddb-4ec8-867a-9002952a8058","Type":"ContainerDied","Data":"6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8"}
Sep 11 16:30:03.546786 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.546704    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-bqcrn" event={"ID":"714e8623-bddb-4ec8-867a-9002952a8058","Type":"ContainerDied","Data":"6635f1ae17374a046ee27fdf6b9a8571eddb38105e254345603ccb2d79d1d2b3"}
Sep 11 16:30:03.546786 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.546732    2550 scope.go:117] "RemoveContainer" containerID="6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8"
Sep 11 16:30:03.561749 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.561724    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-bqcrn"]
Sep 11 16:30:03.565420 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.565396    2550 scope.go:117] "RemoveContainer" containerID="6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8"
Sep 11 16:30:03.565902 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:30:03.565882    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8\": container with ID starting with 6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8 not found: ID does not exist" containerID="6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8"
Sep 11 16:30:03.565986 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.565909    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8"} err="failed to get container status \"6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8\": rpc error: code = NotFound desc = could not find container \"6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8\": container with ID starting with 6e61875f08f4b066f8597ba46a9e49d1b0f195440168aa03d42c7a32820b4fd8 not found: ID does not exist"
Sep 11 16:30:03.566035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:03.566008    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-656b75fc8d-bqcrn"]
Sep 11 16:30:04.552629 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:04.552592    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj" event={"ID":"9b471abb-626e-4430-b15c-57694b24b497","Type":"ContainerStarted","Data":"8981f087fb81cae201705d01a63bd3fc9686744b34d1971841b05254f74d86be"}
Sep 11 16:30:04.553066 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:04.552751    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:04.567843 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:04.567802    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj" podStartSLOduration=1.9624309800000002 podStartE2EDuration="2.567787611s" totalImagesPullingTime="605.356631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:30:04.566304455 +0000 UTC m=+779.522352082" watchObservedRunningTime="2026-09-11 16:30:04.567787611 +0000 UTC m=+779.523835253"
Sep 11 16:30:05.476179 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:05.476152    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714e8623-bddb-4ec8-867a-9002952a8058" path="/var/lib/kubelet/pods/714e8623-bddb-4ec8-867a-9002952a8058/volumes"
Sep 11 16:30:35.563597 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:35.563514    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-656b75fc8d-p8vmj"
Sep 11 16:30:36.862512 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.862477    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-mmb5z"]
Sep 11 16:30:36.863033 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.863011    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714e8623-bddb-4ec8-867a-9002952a8058" containerName="manager"
Sep 11 16:30:36.863112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.863036    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="714e8623-bddb-4ec8-867a-9002952a8058" containerName="manager"
Sep 11 16:30:36.863163 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.863142    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="714e8623-bddb-4ec8-867a-9002952a8058" containerName="manager"
Sep 11 16:30:36.866036 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.866014    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:36.867787 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.867766    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 11 16:30:36.867865 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.867825    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-bqpbh\""
Sep 11 16:30:36.872771 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.872751    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-hdjb4"]
Sep 11 16:30:36.876097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.876078    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-mmb5z"]
Sep 11 16:30:36.876187 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.876151    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:36.878185 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.878158    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 11 16:30:36.878351 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.878201    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 11 16:30:36.878668 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.878649    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-h6jqb\""
Sep 11 16:30:36.886097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.886076    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hdjb4"]
Sep 11 16:30:36.957150 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.957118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9w2d\" (UniqueName: \"kubernetes.io/projected/76bee7ac-7333-47d7-88c3-2fc651e5c92e-kube-api-access-p9w2d\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:36.957276 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.957173    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-cert\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:36.957276 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.957246    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b8e008-b3b9-48e3-a438-2df7cd58d09b-tls-certs\") pod \"model-serving-api-7b954ffc95-mmb5z\" (UID: \"10b8e008-b3b9-48e3-a438-2df7cd58d09b\") " pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:36.957397 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.957314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbn8\" (UniqueName: \"kubernetes.io/projected/10b8e008-b3b9-48e3-a438-2df7cd58d09b-kube-api-access-mxbn8\") pod \"model-serving-api-7b954ffc95-mmb5z\" (UID: \"10b8e008-b3b9-48e3-a438-2df7cd58d09b\") " pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:36.957397 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:36.957344    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-metrics-certs\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.058050 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.058022    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b8e008-b3b9-48e3-a438-2df7cd58d09b-tls-certs\") pod \"model-serving-api-7b954ffc95-mmb5z\" (UID: \"10b8e008-b3b9-48e3-a438-2df7cd58d09b\") " pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:37.058171 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.058068    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbn8\" (UniqueName: \"kubernetes.io/projected/10b8e008-b3b9-48e3-a438-2df7cd58d09b-kube-api-access-mxbn8\") pod \"model-serving-api-7b954ffc95-mmb5z\" (UID: \"10b8e008-b3b9-48e3-a438-2df7cd58d09b\") " pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:37.058240 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.058192    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-metrics-certs\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.058298 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.058245    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9w2d\" (UniqueName: \"kubernetes.io/projected/76bee7ac-7333-47d7-88c3-2fc651e5c92e-kube-api-access-p9w2d\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.058354 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.058296    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-cert\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.058405 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:30:37.058396    2550 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 16:30:37.058480 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:30:37.058467    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-cert podName:76bee7ac-7333-47d7-88c3-2fc651e5c92e nodeName:}" failed. No retries permitted until 2026-09-11 16:30:37.558447494 +0000 UTC m=+812.514495114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-cert") pod "odh-model-controller-7b966d7984-hdjb4" (UID: "76bee7ac-7333-47d7-88c3-2fc651e5c92e") : secret "odh-model-controller-webhook-cert" not found
Sep 11 16:30:37.060554 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.060517    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10b8e008-b3b9-48e3-a438-2df7cd58d09b-tls-certs\") pod \"model-serving-api-7b954ffc95-mmb5z\" (UID: \"10b8e008-b3b9-48e3-a438-2df7cd58d09b\") " pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:37.060554 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.060536    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-metrics-certs\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.068175 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.068148    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbn8\" (UniqueName: \"kubernetes.io/projected/10b8e008-b3b9-48e3-a438-2df7cd58d09b-kube-api-access-mxbn8\") pod \"model-serving-api-7b954ffc95-mmb5z\" (UID: \"10b8e008-b3b9-48e3-a438-2df7cd58d09b\") " pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:37.068263 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.068174    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9w2d\" (UniqueName: \"kubernetes.io/projected/76bee7ac-7333-47d7-88c3-2fc651e5c92e-kube-api-access-p9w2d\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.176331 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.176268    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:37.447964 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.447938    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-mmb5z"]
Sep 11 16:30:37.449496 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:30:37.449470    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b8e008_b3b9_48e3_a438_2df7cd58d09b.slice/crio-71d95f1b1cc38cec16576018bcabc922ea14a1a103f1928179e1476066f16d0e WatchSource:0}: Error finding container 71d95f1b1cc38cec16576018bcabc922ea14a1a103f1928179e1476066f16d0e: Status 404 returned error can't find the container with id 71d95f1b1cc38cec16576018bcabc922ea14a1a103f1928179e1476066f16d0e
Sep 11 16:30:37.562921 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.562891    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-cert\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.565139 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.565120    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/76bee7ac-7333-47d7-88c3-2fc651e5c92e-cert\") pod \"odh-model-controller-7b966d7984-hdjb4\" (UID: \"76bee7ac-7333-47d7-88c3-2fc651e5c92e\") " pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:37.704320 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.704264    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-mmb5z" event={"ID":"10b8e008-b3b9-48e3-a438-2df7cd58d09b","Type":"ContainerStarted","Data":"71d95f1b1cc38cec16576018bcabc922ea14a1a103f1928179e1476066f16d0e"}
Sep 11 16:30:37.786168 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:37.786146    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:38.055020 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:38.054994    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hdjb4"]
Sep 11 16:30:38.056821 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:30:38.056793    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bee7ac_7333_47d7_88c3_2fc651e5c92e.slice/crio-414dea5e8268ea663b00a99973c9c55f392eebe356fda42b545fcc518eca6222 WatchSource:0}: Error finding container 414dea5e8268ea663b00a99973c9c55f392eebe356fda42b545fcc518eca6222: Status 404 returned error can't find the container with id 414dea5e8268ea663b00a99973c9c55f392eebe356fda42b545fcc518eca6222
Sep 11 16:30:38.711228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:38.711193    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hdjb4" event={"ID":"76bee7ac-7333-47d7-88c3-2fc651e5c92e","Type":"ContainerStarted","Data":"414dea5e8268ea663b00a99973c9c55f392eebe356fda42b545fcc518eca6222"}
Sep 11 16:30:39.717918 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:39.717879    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-mmb5z" event={"ID":"10b8e008-b3b9-48e3-a438-2df7cd58d09b","Type":"ContainerStarted","Data":"8b9b52b7ce4d5daca2ea961f9998842af5459c51622293fa0676c4d73b7a65e4"}
Sep 11 16:30:39.718367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:39.717980    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:39.734817 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:39.734756    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-mmb5z" podStartSLOduration=2.226921081 podStartE2EDuration="3.734736902s" totalImagesPullingTime="1.507815821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:30:39.733001969 +0000 UTC m=+814.689049599" watchObservedRunningTime="2026-09-11 16:30:39.734736902 +0000 UTC m=+814.690784531"
Sep 11 16:30:41.730265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:41.730228    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hdjb4" event={"ID":"76bee7ac-7333-47d7-88c3-2fc651e5c92e","Type":"ContainerStarted","Data":"90bd7f55dd066c326b9ae86d3d79c06b495de62d8be6ba7e3d509cddc971a5d9"}
Sep 11 16:30:41.730690 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:41.730370    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:30:41.745877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:41.745829    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-hdjb4" podStartSLOduration=2.568457254 podStartE2EDuration="5.7458156s" totalImagesPullingTime="3.177358346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:30:41.745395531 +0000 UTC m=+816.701443159" watchObservedRunningTime="2026-09-11 16:30:41.7458156 +0000 UTC m=+816.701863228"
Sep 11 16:30:50.728122 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:50.728089    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-mmb5z"
Sep 11 16:30:52.737009 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:30:52.736981    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-hdjb4"
Sep 11 16:32:05.455535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:05.455461    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:32:05.458034 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:05.458013    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:32:33.139836 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.139804    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"]
Sep 11 16:32:33.142908 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.142887    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.144986 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.144965    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 11 16:32:33.145459 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.145440    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 11 16:32:33.145621 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.145496    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-f6kjc\""
Sep 11 16:32:33.145800 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.145776    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 11 16:32:33.145908 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.145831    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-wp7q7\""
Sep 11 16:32:33.151519 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.151496    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"]
Sep 11 16:32:33.178405 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.178383    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dde8b4-9920-43fa-b3ff-a1de91288d99-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.178521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.178413    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmvfj\" (UniqueName: \"kubernetes.io/projected/20dde8b4-9920-43fa-b3ff-a1de91288d99-kube-api-access-qmvfj\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.279325 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.279296    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dde8b4-9920-43fa-b3ff-a1de91288d99-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.279325 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.279327    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmvfj\" (UniqueName: \"kubernetes.io/projected/20dde8b4-9920-43fa-b3ff-a1de91288d99-kube-api-access-qmvfj\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.281660 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.281640    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dde8b4-9920-43fa-b3ff-a1de91288d99-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.288108 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.288088    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmvfj\" (UniqueName: \"kubernetes.io/projected/20dde8b4-9920-43fa-b3ff-a1de91288d99-kube-api-access-qmvfj\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.455111 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.455036    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:33.726438 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:33.726412    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"]
Sep 11 16:32:33.728436 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:32:33.728413    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dde8b4_9920_43fa_b3ff_a1de91288d99.slice/crio-bdb4f0e75613b80fba9f1a77e426587da08ee5886f2fb217b7180427b380dbc5 WatchSource:0}: Error finding container bdb4f0e75613b80fba9f1a77e426587da08ee5886f2fb217b7180427b380dbc5: Status 404 returned error can't find the container with id bdb4f0e75613b80fba9f1a77e426587da08ee5886f2fb217b7180427b380dbc5
Sep 11 16:32:34.261728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:34.261692    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" event={"ID":"20dde8b4-9920-43fa-b3ff-a1de91288d99","Type":"ContainerStarted","Data":"bdb4f0e75613b80fba9f1a77e426587da08ee5886f2fb217b7180427b380dbc5"}
Sep 11 16:32:37.282831 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:37.282794    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" event={"ID":"20dde8b4-9920-43fa-b3ff-a1de91288d99","Type":"ContainerStarted","Data":"a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495"}
Sep 11 16:32:37.283230 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:37.282922    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:32:37.301018 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:32:37.300975    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podStartSLOduration=1.780674001 podStartE2EDuration="4.300961452s" totalImagesPullingTime="2.520287451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:37.298728594 +0000 UTC m=+932.254776223" watchObservedRunningTime="2026-09-11 16:32:37.300961452 +0000 UTC m=+932.257009081"
Sep 11 16:33:08.290089 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:08.290061    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:33:14.745683 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:14.745642    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"]
Sep 11 16:33:18.290130 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:18.290088    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:33:28.289579 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:28.289491    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:33:29.769393 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:29.769329    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main" containerID="cri-o://a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495" gracePeriod=45
Sep 11 16:33:38.289211 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:38.289172    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:33:38.289593 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:38.289285    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:33:48.289149 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:48.289112    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:33:58.289400 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:33:58.289364    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:34:00.013530 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.013507    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:34:00.095962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.095934    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmvfj\" (UniqueName: \"kubernetes.io/projected/20dde8b4-9920-43fa-b3ff-a1de91288d99-kube-api-access-qmvfj\") pod \"20dde8b4-9920-43fa-b3ff-a1de91288d99\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") "
Sep 11 16:34:00.096070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.096014    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dde8b4-9920-43fa-b3ff-a1de91288d99-tls-certs\") pod \"20dde8b4-9920-43fa-b3ff-a1de91288d99\" (UID: \"20dde8b4-9920-43fa-b3ff-a1de91288d99\") "
Sep 11 16:34:00.098092 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.098064    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dde8b4-9920-43fa-b3ff-a1de91288d99-kube-api-access-qmvfj" (OuterVolumeSpecName: "kube-api-access-qmvfj") pod "20dde8b4-9920-43fa-b3ff-a1de91288d99" (UID: "20dde8b4-9920-43fa-b3ff-a1de91288d99"). InnerVolumeSpecName "kube-api-access-qmvfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:34:00.098195 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.098158    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20dde8b4-9920-43fa-b3ff-a1de91288d99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20dde8b4-9920-43fa-b3ff-a1de91288d99" (UID: "20dde8b4-9920-43fa-b3ff-a1de91288d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:34:00.196401 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.196372    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dde8b4-9920-43fa-b3ff-a1de91288d99-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:34:00.196401 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.196397    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmvfj\" (UniqueName: \"kubernetes.io/projected/20dde8b4-9920-43fa-b3ff-a1de91288d99-kube-api-access-qmvfj\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:34:00.670222 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.670187    2550 generic.go:356] "Generic (PLEG): container finished" podID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerID="a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495" exitCode=0
Sep 11 16:34:00.670396 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.670235    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"
Sep 11 16:34:00.670396 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.670273    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" event={"ID":"20dde8b4-9920-43fa-b3ff-a1de91288d99","Type":"ContainerDied","Data":"a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495"}
Sep 11 16:34:00.670396 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.670328    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9" event={"ID":"20dde8b4-9920-43fa-b3ff-a1de91288d99","Type":"ContainerDied","Data":"bdb4f0e75613b80fba9f1a77e426587da08ee5886f2fb217b7180427b380dbc5"}
Sep 11 16:34:00.670396 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.670376    2550 scope.go:117] "RemoveContainer" containerID="a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495"
Sep 11 16:34:00.682203 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.682179    2550 scope.go:117] "RemoveContainer" containerID="a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495"
Sep 11 16:34:00.682519 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:34:00.682498    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495\": container with ID starting with a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495 not found: ID does not exist" containerID="a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495"
Sep 11 16:34:00.682611 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.682527    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495"} err="failed to get container status \"a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495\": rpc error: code = NotFound desc = could not find container \"a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495\": container with ID starting with a8743b2802a0e8e40c97a243f11b5e655ccb44fbe24199f11f62542cdedfe495 not found: ID does not exist"
Sep 11 16:34:00.695001 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.694976    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"]
Sep 11 16:34:00.698402 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:00.698382    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-ds5q9"]
Sep 11 16:34:01.476497 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:01.476463    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" path="/var/lib/kubelet/pods/20dde8b4-9920-43fa-b3ff-a1de91288d99/volumes"
Sep 11 16:34:10.185618 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.185580    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"]
Sep 11 16:34:10.185992 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.185910    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main"
Sep 11 16:34:10.185992 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.185922    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main"
Sep 11 16:34:10.185992 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.185976    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="20dde8b4-9920-43fa-b3ff-a1de91288d99" containerName="main"
Sep 11 16:34:10.190307 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.190285    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.193074 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.193049    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-mznvr\""
Sep 11 16:34:10.193191 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.193109    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 11 16:34:10.193191 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.193050    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 11 16:34:10.193191 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.193051    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-brd2q\""
Sep 11 16:34:10.193357 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.193233    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 11 16:34:10.200564 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.200529    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"]
Sep 11 16:34:10.268047 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.268025    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfvq\" (UniqueName: \"kubernetes.io/projected/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-kube-api-access-mrfvq\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.268165 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.268067    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.368741 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.368706    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfvq\" (UniqueName: \"kubernetes.io/projected/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-kube-api-access-mrfvq\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.368868 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.368786    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.371082 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.371059    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.375785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.375763    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfvq\" (UniqueName: \"kubernetes.io/projected/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-kube-api-access-mrfvq\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.501141 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.501083    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:10.777263 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:10.777238    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"]
Sep 11 16:34:10.778891 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:34:10.778864    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cd86e0_6b59_4a3c_a42d_07717fd1c1db.slice/crio-7de9f6b6867005cf0efd60e80d7aa77b4b8780f81942c1650022610f4be182b7 WatchSource:0}: Error finding container 7de9f6b6867005cf0efd60e80d7aa77b4b8780f81942c1650022610f4be182b7: Status 404 returned error can't find the container with id 7de9f6b6867005cf0efd60e80d7aa77b4b8780f81942c1650022610f4be182b7
Sep 11 16:34:11.729859 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:11.729824    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" event={"ID":"76cd86e0-6b59-4a3c-a42d-07717fd1c1db","Type":"ContainerStarted","Data":"5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee"}
Sep 11 16:34:11.730279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:11.729873    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" event={"ID":"76cd86e0-6b59-4a3c-a42d-07717fd1c1db","Type":"ContainerStarted","Data":"7de9f6b6867005cf0efd60e80d7aa77b4b8780f81942c1650022610f4be182b7"}
Sep 11 16:34:11.730279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:11.729938    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:42.735860 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:42.735833    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:34:42.752323 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:42.752278    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podStartSLOduration=32.752265563 podStartE2EDuration="32.752265563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:34:11.749773166 +0000 UTC m=+1026.705820794" watchObservedRunningTime="2026-09-11 16:34:42.752265563 +0000 UTC m=+1057.708313191"
Sep 11 16:34:49.568165 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:49.568124    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"]
Sep 11 16:34:52.736035 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:34:52.735995    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:35:02.736328 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:02.736250    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:35:04.587106 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:04.587039    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main" containerID="cri-o://5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee" gracePeriod=45
Sep 11 16:35:12.736134 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:12.736098    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:35:12.736523 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:12.736213    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:35:22.736180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:22.736147    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:35:32.736046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:32.736009    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:35:34.833090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.833063    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:35:34.876788 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.876722    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-tls-certs\") pod \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") "
Sep 11 16:35:34.876910 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.876801    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrfvq\" (UniqueName: \"kubernetes.io/projected/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-kube-api-access-mrfvq\") pod \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\" (UID: \"76cd86e0-6b59-4a3c-a42d-07717fd1c1db\") "
Sep 11 16:35:34.878905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.878870    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76cd86e0-6b59-4a3c-a42d-07717fd1c1db" (UID: "76cd86e0-6b59-4a3c-a42d-07717fd1c1db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:35:34.879280 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.878918    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-kube-api-access-mrfvq" (OuterVolumeSpecName: "kube-api-access-mrfvq") pod "76cd86e0-6b59-4a3c-a42d-07717fd1c1db" (UID: "76cd86e0-6b59-4a3c-a42d-07717fd1c1db"). InnerVolumeSpecName "kube-api-access-mrfvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:35:34.977861 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.977829    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrfvq\" (UniqueName: \"kubernetes.io/projected/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-kube-api-access-mrfvq\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:35:34.977861 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:34.977856    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cd86e0-6b59-4a3c-a42d-07717fd1c1db-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:35:35.128143 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.128070    2550 generic.go:356] "Generic (PLEG): container finished" podID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerID="5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee" exitCode=0
Sep 11 16:35:35.128143 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.128113    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"
Sep 11 16:35:35.128303 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.128152    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" event={"ID":"76cd86e0-6b59-4a3c-a42d-07717fd1c1db","Type":"ContainerDied","Data":"5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee"}
Sep 11 16:35:35.128303 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.128201    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l" event={"ID":"76cd86e0-6b59-4a3c-a42d-07717fd1c1db","Type":"ContainerDied","Data":"7de9f6b6867005cf0efd60e80d7aa77b4b8780f81942c1650022610f4be182b7"}
Sep 11 16:35:35.128303 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.128224    2550 scope.go:117] "RemoveContainer" containerID="5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee"
Sep 11 16:35:35.137706 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.137624    2550 scope.go:117] "RemoveContainer" containerID="5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee"
Sep 11 16:35:35.139136 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:35:35.139112    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee\": container with ID starting with 5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee not found: ID does not exist" containerID="5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee"
Sep 11 16:35:35.139244 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.139145    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee"} err="failed to get container status \"5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee\": rpc error: code = NotFound desc = could not find container \"5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee\": container with ID starting with 5258c50f63e7d92f9e7af473fe5e5f5582341a9c539813f5b39e71f2df1301ee not found: ID does not exist"
Sep 11 16:35:35.150494 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.150475    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"]
Sep 11 16:35:35.154614 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.154593    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-shm5l"]
Sep 11 16:35:35.478590 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:35:35.478502    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" path="/var/lib/kubelet/pods/76cd86e0-6b59-4a3c-a42d-07717fd1c1db/volumes"
Sep 11 16:36:03.551845 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.551814    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"]
Sep 11 16:36:03.552301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.552114    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main"
Sep 11 16:36:03.552301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.552125    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main"
Sep 11 16:36:03.552301 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.552187    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cd86e0-6b59-4a3c-a42d-07717fd1c1db" containerName="main"
Sep 11 16:36:03.555013 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.554992    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.557297 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.557274    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 11 16:36:03.557416 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.557390    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 11 16:36:03.557834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.557816    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 11 16:36:03.557935 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.557835    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-nwwlh\""
Sep 11 16:36:03.558072 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.558059    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-st64d\""
Sep 11 16:36:03.586095 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.586068    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.586207 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.586127    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwrl\" (UniqueName: \"kubernetes.io/projected/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-kube-api-access-wxwrl\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.586977 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.586958    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"]
Sep 11 16:36:03.686869 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.686835    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.687021 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.686877    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwrl\" (UniqueName: \"kubernetes.io/projected/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-kube-api-access-wxwrl\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.689212 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.689188    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.694682 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.694660    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwrl\" (UniqueName: \"kubernetes.io/projected/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-kube-api-access-wxwrl\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:03.865064 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:03.865042    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:04.135963 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:04.135941    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"]
Sep 11 16:36:04.138209 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:36:04.138178    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68657611_f5d6_47b7_8cb4_e46bd7a3a7b1.slice/crio-45161cb068d77cb395ccf54c6659ee472e96756f5567e4e2275635266a11076e WatchSource:0}: Error finding container 45161cb068d77cb395ccf54c6659ee472e96756f5567e4e2275635266a11076e: Status 404 returned error can't find the container with id 45161cb068d77cb395ccf54c6659ee472e96756f5567e4e2275635266a11076e
Sep 11 16:36:04.263956 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:04.263915    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" event={"ID":"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1","Type":"ContainerStarted","Data":"d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587"}
Sep 11 16:36:04.264131 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:04.263967    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" event={"ID":"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1","Type":"ContainerStarted","Data":"45161cb068d77cb395ccf54c6659ee472e96756f5567e4e2275635266a11076e"}
Sep 11 16:36:04.264131 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:04.264109    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:04.281451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:04.281408    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podStartSLOduration=1.281393818 podStartE2EDuration="1.281393818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:04.279912529 +0000 UTC m=+1139.235960182" watchObservedRunningTime="2026-09-11 16:36:04.281393818 +0000 UTC m=+1139.237441446"
Sep 11 16:36:35.270835 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:35.270748    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:36:47.392768 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.392730    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"]
Sep 11 16:36:47.399107 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.399068    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.405562 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.405518    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 11 16:36:47.406023 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.405998    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 11 16:36:47.406260 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.406241    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 11 16:36:47.406718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.406696    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-zdtmz\""
Sep 11 16:36:47.408047 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.408024    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"]
Sep 11 16:36:47.497113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.497084    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3ae13f-f2c1-4e91-98e4-92375f175992-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.497113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.497114    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f697s\" (UniqueName: \"kubernetes.io/projected/7f3ae13f-f2c1-4e91-98e4-92375f175992-kube-api-access-f697s\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.497294 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.497143    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.497294 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.497189    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.497294 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.497237    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.497294 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.497262    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598084 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598049    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598250 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598092    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598250 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598154    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598250 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598183    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598250 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598228    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3ae13f-f2c1-4e91-98e4-92375f175992-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598252    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f697s\" (UniqueName: \"kubernetes.io/projected/7f3ae13f-f2c1-4e91-98e4-92375f175992-kube-api-access-f697s\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598587 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598538    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598735 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598627    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.598735 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.598637    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.600461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.600443    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.600629 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.600609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3ae13f-f2c1-4e91-98e4-92375f175992-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.604955 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.604935    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f697s\" (UniqueName: \"kubernetes.io/projected/7f3ae13f-f2c1-4e91-98e4-92375f175992-kube-api-access-f697s\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.714639 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.714583    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:47.987905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.987821    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"]
Sep 11 16:36:47.990914 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:36:47.990883    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3ae13f_f2c1_4e91_98e4_92375f175992.slice/crio-4ca9d6465e2077dbf56d0fffc036de127e50cb9df38b5979082f5a6a9f8cfddc WatchSource:0}: Error finding container 4ca9d6465e2077dbf56d0fffc036de127e50cb9df38b5979082f5a6a9f8cfddc: Status 404 returned error can't find the container with id 4ca9d6465e2077dbf56d0fffc036de127e50cb9df38b5979082f5a6a9f8cfddc
Sep 11 16:36:47.992848 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:47.992828    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:36:48.473371 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:48.473335    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" event={"ID":"7f3ae13f-f2c1-4e91-98e4-92375f175992","Type":"ContainerStarted","Data":"4ca9d6465e2077dbf56d0fffc036de127e50cb9df38b5979082f5a6a9f8cfddc"}
Sep 11 16:36:50.485614 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:50.485573    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" event={"ID":"7f3ae13f-f2c1-4e91-98e4-92375f175992","Type":"ContainerStarted","Data":"1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"}
Sep 11 16:36:50.501117 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:50.500914    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" podStartSLOduration=1.981887545 podStartE2EDuration="3.500899933s" totalImagesPullingTime="1.519012388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:50.500639876 +0000 UTC m=+1185.456687533" watchObservedRunningTime="2026-09-11 16:36:50.500899933 +0000 UTC m=+1185.456947561"
Sep 11 16:36:50.715514 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:50.715482    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:54.683693 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:54.683658    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"]
Sep 11 16:36:54.692976 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:36:54.692937    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:36:54.692976 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:36:54.692976 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:36:54.692976 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093" cmd=["/bin/sleep","15"]
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:54.692993    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" message="" err=<
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:36:54.693005    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" podUID="7f3ae13f-f2c1-4e91-98e4-92375f175992" containerName="main" containerID="cri-o://1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"
Sep 11 16:36:54.693209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:54.693038    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" podUID="7f3ae13f-f2c1-4e91-98e4-92375f175992" containerName="main" containerID="cri-o://1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093" gracePeriod=60
Sep 11 16:36:54.935403 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:54.935349    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:55.058203 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058175    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f697s\" (UniqueName: \"kubernetes.io/projected/7f3ae13f-f2c1-4e91-98e4-92375f175992-kube-api-access-f697s\") pod \"7f3ae13f-f2c1-4e91-98e4-92375f175992\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") "
Sep 11 16:36:55.058203 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058207    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-tmp-dir\") pod \"7f3ae13f-f2c1-4e91-98e4-92375f175992\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") "
Sep 11 16:36:55.058377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058232    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-dshm\") pod \"7f3ae13f-f2c1-4e91-98e4-92375f175992\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") "
Sep 11 16:36:55.058377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058293    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-home\") pod \"7f3ae13f-f2c1-4e91-98e4-92375f175992\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") "
Sep 11 16:36:55.058377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058320    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3ae13f-f2c1-4e91-98e4-92375f175992-tls-certs\") pod \"7f3ae13f-f2c1-4e91-98e4-92375f175992\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") "
Sep 11 16:36:55.058519 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058378    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-model-cache\") pod \"7f3ae13f-f2c1-4e91-98e4-92375f175992\" (UID: \"7f3ae13f-f2c1-4e91-98e4-92375f175992\") "
Sep 11 16:36:55.058519 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058464    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f3ae13f-f2c1-4e91-98e4-92375f175992" (UID: "7f3ae13f-f2c1-4e91-98e4-92375f175992"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:55.058655 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058538    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-home" (OuterVolumeSpecName: "home") pod "7f3ae13f-f2c1-4e91-98e4-92375f175992" (UID: "7f3ae13f-f2c1-4e91-98e4-92375f175992"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:55.058711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058635    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:36:55.058711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058678    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:36:55.058802 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.058709    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f3ae13f-f2c1-4e91-98e4-92375f175992" (UID: "7f3ae13f-f2c1-4e91-98e4-92375f175992"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:55.060500 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.060478    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-dshm" (OuterVolumeSpecName: "dshm") pod "7f3ae13f-f2c1-4e91-98e4-92375f175992" (UID: "7f3ae13f-f2c1-4e91-98e4-92375f175992"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:36:55.060591 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.060481    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3ae13f-f2c1-4e91-98e4-92375f175992-kube-api-access-f697s" (OuterVolumeSpecName: "kube-api-access-f697s") pod "7f3ae13f-f2c1-4e91-98e4-92375f175992" (UID: "7f3ae13f-f2c1-4e91-98e4-92375f175992"). InnerVolumeSpecName "kube-api-access-f697s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:36:55.060591 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.060576    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3ae13f-f2c1-4e91-98e4-92375f175992-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f3ae13f-f2c1-4e91-98e4-92375f175992" (UID: "7f3ae13f-f2c1-4e91-98e4-92375f175992"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:36:55.159562 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.159518    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3ae13f-f2c1-4e91-98e4-92375f175992-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:36:55.159562 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.159554    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:36:55.159562 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.159565    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f697s\" (UniqueName: \"kubernetes.io/projected/7f3ae13f-f2c1-4e91-98e4-92375f175992-kube-api-access-f697s\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:36:55.159740 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.159574    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3ae13f-f2c1-4e91-98e4-92375f175992-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:36:55.514504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.514473    2550 generic.go:356] "Generic (PLEG): container finished" podID="7f3ae13f-f2c1-4e91-98e4-92375f175992" containerID="1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093" exitCode=0
Sep 11 16:36:55.514673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.514522    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"
Sep 11 16:36:55.514673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.514565    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" event={"ID":"7f3ae13f-f2c1-4e91-98e4-92375f175992","Type":"ContainerDied","Data":"1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"}
Sep 11 16:36:55.514673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.514616    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn" event={"ID":"7f3ae13f-f2c1-4e91-98e4-92375f175992","Type":"ContainerDied","Data":"4ca9d6465e2077dbf56d0fffc036de127e50cb9df38b5979082f5a6a9f8cfddc"}
Sep 11 16:36:55.514673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.514649    2550 scope.go:117] "RemoveContainer" containerID="1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"
Sep 11 16:36:55.523314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.523295    2550 scope.go:117] "RemoveContainer" containerID="1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"
Sep 11 16:36:55.523594 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:36:55.523574    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093\": container with ID starting with 1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093 not found: ID does not exist" containerID="1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"
Sep 11 16:36:55.523666 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.523604    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093"} err="failed to get container status \"1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093\": rpc error: code = NotFound desc = could not find container \"1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093\": container with ID starting with 1e09ccde3db21ecab39c97d952f29dcafc74b1fc2b92ecd9683ac98719939093 not found: ID does not exist"
Sep 11 16:36:55.528347 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.528324    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"]
Sep 11 16:36:55.531733 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:55.531712    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-gfmwn"]
Sep 11 16:36:57.477341 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:36:57.477261    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3ae13f-f2c1-4e91-98e4-92375f175992" path="/var/lib/kubelet/pods/7f3ae13f-f2c1-4e91-98e4-92375f175992/volumes"
Sep 11 16:37:05.495042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:05.495007    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:37:05.497512 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:05.497488    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:37:12.881703 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.881673    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"]
Sep 11 16:37:12.882223 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.882182    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3ae13f-f2c1-4e91-98e4-92375f175992" containerName="main"
Sep 11 16:37:12.882223 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.882206    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3ae13f-f2c1-4e91-98e4-92375f175992" containerName="main"
Sep 11 16:37:12.882381 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.882312    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3ae13f-f2c1-4e91-98e4-92375f175992" containerName="main"
Sep 11 16:37:12.886521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.886501    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:12.888995 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.888969    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 11 16:37:12.888995 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.888976    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 11 16:37:12.889170 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.889155    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 11 16:37:12.889613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.889598    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-w56tq\""
Sep 11 16:37:12.895203 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.895183    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"]
Sep 11 16:37:12.986848 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.986816    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:12.987011 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.986852    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:12.987011 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.986886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499q7\" (UniqueName: \"kubernetes.io/projected/4a563739-3356-4e1e-9600-f36125f8c486-kube-api-access-499q7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:12.987011 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.986964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:12.987011 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.987007    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:12.987158 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:12.987028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a563739-3356-4e1e-9600-f36125f8c486-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.087535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.087507    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.087685 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.087579    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499q7\" (UniqueName: \"kubernetes.io/projected/4a563739-3356-4e1e-9600-f36125f8c486-kube-api-access-499q7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.087685 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.087615    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.087685 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.087636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.087685 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.087655    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a563739-3356-4e1e-9600-f36125f8c486-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.087853 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.087716    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.088048 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.088030    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.088103 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.088062    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.088103 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.088095    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.089858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.089838    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.090135 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.090115    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a563739-3356-4e1e-9600-f36125f8c486-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.098315 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.098291    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-499q7\" (UniqueName: \"kubernetes.io/projected/4a563739-3356-4e1e-9600-f36125f8c486-kube-api-access-499q7\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.197683 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.197629    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:13.498744 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:37:13.498716    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a563739_3356_4e1e_9600_f36125f8c486.slice/crio-af9318aa5addf2e82868a1d635c6f5f30df10326ea8137c7754eabaadde16319 WatchSource:0}: Error finding container af9318aa5addf2e82868a1d635c6f5f30df10326ea8137c7754eabaadde16319: Status 404 returned error can't find the container with id af9318aa5addf2e82868a1d635c6f5f30df10326ea8137c7754eabaadde16319
Sep 11 16:37:13.499964 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.499936    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"]
Sep 11 16:37:13.600082 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.600050    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" event={"ID":"4a563739-3356-4e1e-9600-f36125f8c486","Type":"ContainerStarted","Data":"5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"}
Sep 11 16:37:13.600219 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.600117    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" event={"ID":"4a563739-3356-4e1e-9600-f36125f8c486","Type":"ContainerStarted","Data":"af9318aa5addf2e82868a1d635c6f5f30df10326ea8137c7754eabaadde16319"}
Sep 11 16:37:13.636844 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:13.636797    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" podStartSLOduration=1.636781217 podStartE2EDuration="1.636781217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:37:13.632246984 +0000 UTC m=+1208.588294611" watchObservedRunningTime="2026-09-11 16:37:13.636781217 +0000 UTC m=+1208.592828845"
Sep 11 16:37:14.198773 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:14.198740    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:16.622445 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:16.622415    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"]
Sep 11 16:37:16.631042 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:37:16.631011    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:37:16.631042 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:37:16.631042 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:37:16.631042 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2" cmd=["/bin/sleep","15"]
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:16.631055    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" message="" err=<
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:37:16.631068    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" podUID="4a563739-3356-4e1e-9600-f36125f8c486" containerName="main" containerID="cri-o://5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"
Sep 11 16:37:16.631272 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:16.631111    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" podUID="4a563739-3356-4e1e-9600-f36125f8c486" containerName="main" containerID="cri-o://5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2" gracePeriod=60
Sep 11 16:37:16.877342 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:16.877279    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:17.017469 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017439    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-model-cache\") pod \"4a563739-3356-4e1e-9600-f36125f8c486\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") "
Sep 11 16:37:17.017634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017478    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-home\") pod \"4a563739-3356-4e1e-9600-f36125f8c486\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") "
Sep 11 16:37:17.017634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017504    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-tmp-dir\") pod \"4a563739-3356-4e1e-9600-f36125f8c486\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") "
Sep 11 16:37:17.017634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017565    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a563739-3356-4e1e-9600-f36125f8c486-tls-certs\") pod \"4a563739-3356-4e1e-9600-f36125f8c486\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") "
Sep 11 16:37:17.017634 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017620    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-dshm\") pod \"4a563739-3356-4e1e-9600-f36125f8c486\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") "
Sep 11 16:37:17.017872 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017663    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-499q7\" (UniqueName: \"kubernetes.io/projected/4a563739-3356-4e1e-9600-f36125f8c486-kube-api-access-499q7\") pod \"4a563739-3356-4e1e-9600-f36125f8c486\" (UID: \"4a563739-3356-4e1e-9600-f36125f8c486\") "
Sep 11 16:37:17.017872 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017751    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a563739-3356-4e1e-9600-f36125f8c486" (UID: "4a563739-3356-4e1e-9600-f36125f8c486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:17.017872 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017772    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-home" (OuterVolumeSpecName: "home") pod "4a563739-3356-4e1e-9600-f36125f8c486" (UID: "4a563739-3356-4e1e-9600-f36125f8c486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:17.018010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017884    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a563739-3356-4e1e-9600-f36125f8c486" (UID: "4a563739-3356-4e1e-9600-f36125f8c486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:17.018010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017962    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:37:17.018010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017981    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:37:17.018010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.017995    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:37:17.019858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.019828    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-dshm" (OuterVolumeSpecName: "dshm") pod "4a563739-3356-4e1e-9600-f36125f8c486" (UID: "4a563739-3356-4e1e-9600-f36125f8c486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:37:17.019949 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.019930    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a563739-3356-4e1e-9600-f36125f8c486-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a563739-3356-4e1e-9600-f36125f8c486" (UID: "4a563739-3356-4e1e-9600-f36125f8c486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:37:17.020019 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.019987    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a563739-3356-4e1e-9600-f36125f8c486-kube-api-access-499q7" (OuterVolumeSpecName: "kube-api-access-499q7") pod "4a563739-3356-4e1e-9600-f36125f8c486" (UID: "4a563739-3356-4e1e-9600-f36125f8c486"). InnerVolumeSpecName "kube-api-access-499q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:37:17.118558 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.118512    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a563739-3356-4e1e-9600-f36125f8c486-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:37:17.118558 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.118539    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-499q7\" (UniqueName: \"kubernetes.io/projected/4a563739-3356-4e1e-9600-f36125f8c486-kube-api-access-499q7\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:37:17.118699 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.118575    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a563739-3356-4e1e-9600-f36125f8c486-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:37:17.621807 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.621768    2550 generic.go:356] "Generic (PLEG): container finished" podID="4a563739-3356-4e1e-9600-f36125f8c486" containerID="5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2" exitCode=0
Sep 11 16:37:17.621996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.621818    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"
Sep 11 16:37:17.621996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.621851    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" event={"ID":"4a563739-3356-4e1e-9600-f36125f8c486","Type":"ContainerDied","Data":"5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"}
Sep 11 16:37:17.621996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.621905    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s" event={"ID":"4a563739-3356-4e1e-9600-f36125f8c486","Type":"ContainerDied","Data":"af9318aa5addf2e82868a1d635c6f5f30df10326ea8137c7754eabaadde16319"}
Sep 11 16:37:17.621996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.621929    2550 scope.go:117] "RemoveContainer" containerID="5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"
Sep 11 16:37:17.630718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.630701    2550 scope.go:117] "RemoveContainer" containerID="5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"
Sep 11 16:37:17.630994 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:37:17.630970    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2\": container with ID starting with 5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2 not found: ID does not exist" containerID="5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"
Sep 11 16:37:17.631039 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.630993    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2"} err="failed to get container status \"5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2\": rpc error: code = NotFound desc = could not find container \"5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2\": container with ID starting with 5f7218cfbef1fffea30987c03516b61862607239fbc96335b9293f737a2a0aa2 not found: ID does not exist"
Sep 11 16:37:17.636166 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.636141    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"]
Sep 11 16:37:17.644657 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:17.644638    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfzk69s"]
Sep 11 16:37:19.476949 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:19.476912    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a563739-3356-4e1e-9600-f36125f8c486" path="/var/lib/kubelet/pods/4a563739-3356-4e1e-9600-f36125f8c486/volumes"
Sep 11 16:37:38.937187 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.937147    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"]
Sep 11 16:37:38.937736 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.937463    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a563739-3356-4e1e-9600-f36125f8c486" containerName="main"
Sep 11 16:37:38.937736 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.937474    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a563739-3356-4e1e-9600-f36125f8c486" containerName="main"
Sep 11 16:37:38.937736 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.937536    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a563739-3356-4e1e-9600-f36125f8c486" containerName="main"
Sep 11 16:37:38.940342 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.940318    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:38.942670 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.942644    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 11 16:37:38.942785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.942678    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 11 16:37:38.943509 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.943482    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-tdq6w\""
Sep 11 16:37:38.943650 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.943626    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-hhjln\""
Sep 11 16:37:38.943734 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.943715    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 11 16:37:38.951887 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.951868    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"]
Sep 11 16:37:38.968406 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.968382    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbv6\" (UniqueName: \"kubernetes.io/projected/6bb9c541-681a-4891-9d36-af4bad150d72-kube-api-access-mhbv6\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:38.968504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:38.968434    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb9c541-681a-4891-9d36-af4bad150d72-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.069190 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.069157    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbv6\" (UniqueName: \"kubernetes.io/projected/6bb9c541-681a-4891-9d36-af4bad150d72-kube-api-access-mhbv6\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.069355 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.069201    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb9c541-681a-4891-9d36-af4bad150d72-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.071711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.071686    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb9c541-681a-4891-9d36-af4bad150d72-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.077336 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.077311    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbv6\" (UniqueName: \"kubernetes.io/projected/6bb9c541-681a-4891-9d36-af4bad150d72-kube-api-access-mhbv6\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.252328 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.252256    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.524490 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.524464    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"]
Sep 11 16:37:39.526360 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:37:39.526323    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb9c541_681a_4891_9d36_af4bad150d72.slice/crio-96fddcb9cea497fbda4dbdf5d561203ca0186b76da4be08f26ed8d7ef1c5c8c7 WatchSource:0}: Error finding container 96fddcb9cea497fbda4dbdf5d561203ca0186b76da4be08f26ed8d7ef1c5c8c7: Status 404 returned error can't find the container with id 96fddcb9cea497fbda4dbdf5d561203ca0186b76da4be08f26ed8d7ef1c5c8c7
Sep 11 16:37:39.740307 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.740272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" event={"ID":"6bb9c541-681a-4891-9d36-af4bad150d72","Type":"ContainerStarted","Data":"f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e"}
Sep 11 16:37:39.740495 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.740330    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" event={"ID":"6bb9c541-681a-4891-9d36-af4bad150d72","Type":"ContainerStarted","Data":"96fddcb9cea497fbda4dbdf5d561203ca0186b76da4be08f26ed8d7ef1c5c8c7"}
Sep 11 16:37:39.740495 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.740369    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:37:39.757439 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:37:39.757389    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" podStartSLOduration=1.757377223 podStartE2EDuration="1.757377223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:37:39.756576008 +0000 UTC m=+1234.712623633" watchObservedRunningTime="2026-09-11 16:37:39.757377223 +0000 UTC m=+1234.713424851"
Sep 11 16:38:10.747760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:10.747728    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:38:12.452458 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:12.452424    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"]
Sep 11 16:38:15.270726 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:15.270691    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:38:25.270887 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:25.270849    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:38:27.472757 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:27.472699    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main" containerID="cri-o://d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587" gracePeriod=45
Sep 11 16:38:35.271204 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:35.271165    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:38:35.271583 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:35.271292    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:38:45.270746 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:45.270711    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:38:55.270741 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:55.270699    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:38:57.709443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.709421    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:38:57.835454 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.835430    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-tls-certs\") pod \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") "
Sep 11 16:38:57.835633 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.835479    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxwrl\" (UniqueName: \"kubernetes.io/projected/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-kube-api-access-wxwrl\") pod \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\" (UID: \"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1\") "
Sep 11 16:38:57.837649 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.837622    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-kube-api-access-wxwrl" (OuterVolumeSpecName: "kube-api-access-wxwrl") pod "68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" (UID: "68657611-f5d6-47b7-8cb4-e46bd7a3a7b1"). InnerVolumeSpecName "kube-api-access-wxwrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:57.837765 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.837648    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" (UID: "68657611-f5d6-47b7-8cb4-e46bd7a3a7b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:57.936403 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.936362    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:38:57.936403 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:57.936399    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxwrl\" (UniqueName: \"kubernetes.io/projected/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1-kube-api-access-wxwrl\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:38:58.112281 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.112189    2550 generic.go:356] "Generic (PLEG): container finished" podID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerID="d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587" exitCode=0
Sep 11 16:38:58.112281 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.112231    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"
Sep 11 16:38:58.112507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.112275    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" event={"ID":"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1","Type":"ContainerDied","Data":"d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587"}
Sep 11 16:38:58.112507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.112334    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w" event={"ID":"68657611-f5d6-47b7-8cb4-e46bd7a3a7b1","Type":"ContainerDied","Data":"45161cb068d77cb395ccf54c6659ee472e96756f5567e4e2275635266a11076e"}
Sep 11 16:38:58.112507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.112363    2550 scope.go:117] "RemoveContainer" containerID="d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587"
Sep 11 16:38:58.121012 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.120992    2550 scope.go:117] "RemoveContainer" containerID="d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587"
Sep 11 16:38:58.121258 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:38:58.121238    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587\": container with ID starting with d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587 not found: ID does not exist" containerID="d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587"
Sep 11 16:38:58.121320 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.121270    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587"} err="failed to get container status \"d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587\": rpc error: code = NotFound desc = could not find container \"d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587\": container with ID starting with d8bd771efab44673600bbc40a285a76b571af9a1eebf67d96a00f3d8cbb53587 not found: ID does not exist"
Sep 11 16:38:58.131857 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.131836    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"]
Sep 11 16:38:58.136170 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:58.136151    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schehk67w"]
Sep 11 16:38:59.476834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:38:59.476801    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" path="/var/lib/kubelet/pods/68657611-f5d6-47b7-8cb4-e46bd7a3a7b1/volumes"
Sep 11 16:39:24.145431 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.145392    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"]
Sep 11 16:39:24.145902 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.145768    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main"
Sep 11 16:39:24.145902 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.145781    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main"
Sep 11 16:39:24.145902 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.145847    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="68657611-f5d6-47b7-8cb4-e46bd7a3a7b1" containerName="main"
Sep 11 16:39:24.150318 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.150292    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.152786 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.152752    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 11 16:39:24.152922 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.152830    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 11 16:39:24.152922 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.152884    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-29lpz\""
Sep 11 16:39:24.153046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.152905    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 11 16:39:24.153046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.152991    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-z9cqr\""
Sep 11 16:39:24.159274 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.159250    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"]
Sep 11 16:39:24.329019 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.328984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14819bb2-341b-4321-b514-7d7df37a4491-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.329178 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.329031    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvd2g\" (UniqueName: \"kubernetes.io/projected/14819bb2-341b-4321-b514-7d7df37a4491-kube-api-access-fvd2g\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.429563 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.429475    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14819bb2-341b-4321-b514-7d7df37a4491-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.429563 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.429534    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvd2g\" (UniqueName: \"kubernetes.io/projected/14819bb2-341b-4321-b514-7d7df37a4491-kube-api-access-fvd2g\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.431876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.431847    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14819bb2-341b-4321-b514-7d7df37a4491-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.437660 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.437640    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvd2g\" (UniqueName: \"kubernetes.io/projected/14819bb2-341b-4321-b514-7d7df37a4491-kube-api-access-fvd2g\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.461912 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.461894    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:24.730779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:24.730754    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"]
Sep 11 16:39:24.732669 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:39:24.732627    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14819bb2_341b_4321_b514_7d7df37a4491.slice/crio-6dbd04b820cf972f821c46d8d261bc8d5e344aa2fa5f40878de43e243e5f3925 WatchSource:0}: Error finding container 6dbd04b820cf972f821c46d8d261bc8d5e344aa2fa5f40878de43e243e5f3925: Status 404 returned error can't find the container with id 6dbd04b820cf972f821c46d8d261bc8d5e344aa2fa5f40878de43e243e5f3925
Sep 11 16:39:25.249131 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:25.249093    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" event={"ID":"14819bb2-341b-4321-b514-7d7df37a4491","Type":"ContainerStarted","Data":"c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579"}
Sep 11 16:39:25.249631 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:25.249148    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" event={"ID":"14819bb2-341b-4321-b514-7d7df37a4491","Type":"ContainerStarted","Data":"6dbd04b820cf972f821c46d8d261bc8d5e344aa2fa5f40878de43e243e5f3925"}
Sep 11 16:39:25.249631 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:25.249229    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:25.267285 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:25.267237    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" podStartSLOduration=1.26722193 podStartE2EDuration="1.26722193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:39:25.264638899 +0000 UTC m=+1340.220686526" watchObservedRunningTime="2026-09-11 16:39:25.26722193 +0000 UTC m=+1340.223269558"
Sep 11 16:39:43.513384 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:43.513350    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"]
Sep 11 16:39:50.747716 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:50.747680    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:39:56.255186 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:56.255158    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:39:57.716496 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:57.716462    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"]
Sep 11 16:39:58.532009 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:39:58.531956    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main" containerID="cri-o://f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e" gracePeriod=45
Sep 11 16:40:00.747923 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:00.747885    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:06.255451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:06.255417    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:10.747712 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:10.747676    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:10.748170 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:10.747786    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:40:12.736270 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:12.736234    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main" containerID="cri-o://c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579" gracePeriod=45
Sep 11 16:40:16.255482 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:16.255448    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:20.747669 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:20.747632    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:26.255247 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:26.255210    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:26.255666 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:26.255310    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:40:28.772312 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.772289    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:40:28.883395 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.883366    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhbv6\" (UniqueName: \"kubernetes.io/projected/6bb9c541-681a-4891-9d36-af4bad150d72-kube-api-access-mhbv6\") pod \"6bb9c541-681a-4891-9d36-af4bad150d72\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") "
Sep 11 16:40:28.883571 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.883412    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb9c541-681a-4891-9d36-af4bad150d72-tls-certs\") pod \"6bb9c541-681a-4891-9d36-af4bad150d72\" (UID: \"6bb9c541-681a-4891-9d36-af4bad150d72\") "
Sep 11 16:40:28.885569 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.885518    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb9c541-681a-4891-9d36-af4bad150d72-kube-api-access-mhbv6" (OuterVolumeSpecName: "kube-api-access-mhbv6") pod "6bb9c541-681a-4891-9d36-af4bad150d72" (UID: "6bb9c541-681a-4891-9d36-af4bad150d72"). InnerVolumeSpecName "kube-api-access-mhbv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:40:28.885695 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.885653    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb9c541-681a-4891-9d36-af4bad150d72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bb9c541-681a-4891-9d36-af4bad150d72" (UID: "6bb9c541-681a-4891-9d36-af4bad150d72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:40:28.984442 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.984407    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb9c541-681a-4891-9d36-af4bad150d72-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:40:28.984442 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:28.984436    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbv6\" (UniqueName: \"kubernetes.io/projected/6bb9c541-681a-4891-9d36-af4bad150d72-kube-api-access-mhbv6\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:40:29.552215 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.552179    2550 generic.go:356] "Generic (PLEG): container finished" podID="6bb9c541-681a-4891-9d36-af4bad150d72" containerID="f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e" exitCode=0
Sep 11 16:40:29.552347 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.552223    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" event={"ID":"6bb9c541-681a-4891-9d36-af4bad150d72","Type":"ContainerDied","Data":"f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e"}
Sep 11 16:40:29.552347 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.552236    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"
Sep 11 16:40:29.552347 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.552272    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p" event={"ID":"6bb9c541-681a-4891-9d36-af4bad150d72","Type":"ContainerDied","Data":"96fddcb9cea497fbda4dbdf5d561203ca0186b76da4be08f26ed8d7ef1c5c8c7"}
Sep 11 16:40:29.552347 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.552301    2550 scope.go:117] "RemoveContainer" containerID="f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e"
Sep 11 16:40:29.560564 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.560534    2550 scope.go:117] "RemoveContainer" containerID="f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e"
Sep 11 16:40:29.560801 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:40:29.560784    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e\": container with ID starting with f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e not found: ID does not exist" containerID="f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e"
Sep 11 16:40:29.560846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.560809    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e"} err="failed to get container status \"f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e\": rpc error: code = NotFound desc = could not find container \"f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e\": container with ID starting with f730deff67fe6ffb576fcb9aeee843a03d43566c3180d2aed53c0d6920af5e6e not found: ID does not exist"
Sep 11 16:40:29.567037 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.567015    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"]
Sep 11 16:40:29.571040 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:29.571018    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8bn5p"]
Sep 11 16:40:31.476452 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:31.476417    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" path="/var/lib/kubelet/pods/6bb9c541-681a-4891-9d36-af4bad150d72/volumes"
Sep 11 16:40:36.255225 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:36.255189    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:40:42.986584 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:42.986556    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:40:43.087527 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.087499    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14819bb2-341b-4321-b514-7d7df37a4491-tls-certs\") pod \"14819bb2-341b-4321-b514-7d7df37a4491\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") "
Sep 11 16:40:43.087672 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.087607    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvd2g\" (UniqueName: \"kubernetes.io/projected/14819bb2-341b-4321-b514-7d7df37a4491-kube-api-access-fvd2g\") pod \"14819bb2-341b-4321-b514-7d7df37a4491\" (UID: \"14819bb2-341b-4321-b514-7d7df37a4491\") "
Sep 11 16:40:43.089557 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.089517    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14819bb2-341b-4321-b514-7d7df37a4491-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14819bb2-341b-4321-b514-7d7df37a4491" (UID: "14819bb2-341b-4321-b514-7d7df37a4491"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:40:43.089557 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.089537    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14819bb2-341b-4321-b514-7d7df37a4491-kube-api-access-fvd2g" (OuterVolumeSpecName: "kube-api-access-fvd2g") pod "14819bb2-341b-4321-b514-7d7df37a4491" (UID: "14819bb2-341b-4321-b514-7d7df37a4491"). InnerVolumeSpecName "kube-api-access-fvd2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:40:43.188520 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.188496    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14819bb2-341b-4321-b514-7d7df37a4491-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:40:43.188520 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.188517    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvd2g\" (UniqueName: \"kubernetes.io/projected/14819bb2-341b-4321-b514-7d7df37a4491-kube-api-access-fvd2g\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:40:43.619645 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.619615    2550 generic.go:356] "Generic (PLEG): container finished" podID="14819bb2-341b-4321-b514-7d7df37a4491" containerID="c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579" exitCode=0
Sep 11 16:40:43.619812 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.619658    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" event={"ID":"14819bb2-341b-4321-b514-7d7df37a4491","Type":"ContainerDied","Data":"c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579"}
Sep 11 16:40:43.619812 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.619694    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf" event={"ID":"14819bb2-341b-4321-b514-7d7df37a4491","Type":"ContainerDied","Data":"6dbd04b820cf972f821c46d8d261bc8d5e344aa2fa5f40878de43e243e5f3925"}
Sep 11 16:40:43.619812 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.619715    2550 scope.go:117] "RemoveContainer" containerID="c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579"
Sep 11 16:40:43.619812 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.619660    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"
Sep 11 16:40:43.627951 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.627933    2550 scope.go:117] "RemoveContainer" containerID="c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579"
Sep 11 16:40:43.628242 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:40:43.628223    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579\": container with ID starting with c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579 not found: ID does not exist" containerID="c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579"
Sep 11 16:40:43.628303 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.628253    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579"} err="failed to get container status \"c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579\": rpc error: code = NotFound desc = could not find container \"c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579\": container with ID starting with c5588e92e6721751a511e9f4fce2610a33b895f9049164c4a580de80eb5f9579 not found: ID does not exist"
Sep 11 16:40:43.634173 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.634152    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"]
Sep 11 16:40:43.640355 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.640332    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cb9ftpf"]
Sep 11 16:40:43.691720 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.691687    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"]
Sep 11 16:40:43.692070 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.692053    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main"
Sep 11 16:40:43.692168 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.692074    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main"
Sep 11 16:40:43.692168 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.692096    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main"
Sep 11 16:40:43.692168 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.692105    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main"
Sep 11 16:40:43.692316 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.692210    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="14819bb2-341b-4321-b514-7d7df37a4491" containerName="main"
Sep 11 16:40:43.692316 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.692226    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb9c541-681a-4891-9d36-af4bad150d72" containerName="main"
Sep 11 16:40:43.696391 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.696372    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:43.698595 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.698558    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 11 16:40:43.698595 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.698571    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-pj45c\""
Sep 11 16:40:43.698753 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.698571    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 11 16:40:43.698753 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.698560    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 11 16:40:43.698753 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.698605    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-s4rx9\""
Sep 11 16:40:43.704188 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.704163    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"]
Sep 11 16:40:43.793896 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.793814    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98ca9c4-cabf-49a1-aa03-46191cde0bce-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:43.793896 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.793876    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2tb\" (UniqueName: \"kubernetes.io/projected/b98ca9c4-cabf-49a1-aa03-46191cde0bce-kube-api-access-ch2tb\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:43.894763 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.894731    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98ca9c4-cabf-49a1-aa03-46191cde0bce-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:43.894912 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.894792    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2tb\" (UniqueName: \"kubernetes.io/projected/b98ca9c4-cabf-49a1-aa03-46191cde0bce-kube-api-access-ch2tb\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:43.897247 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.897225    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98ca9c4-cabf-49a1-aa03-46191cde0bce-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:43.902627 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:43.902608    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2tb\" (UniqueName: \"kubernetes.io/projected/b98ca9c4-cabf-49a1-aa03-46191cde0bce-kube-api-access-ch2tb\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:44.006416 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:44.006368    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:44.282883 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:44.282859    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"]
Sep 11 16:40:44.284606 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:40:44.284579    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98ca9c4_cabf_49a1_aa03_46191cde0bce.slice/crio-d1d099edbbf4d3fed1cfa850998fcbc526a836d6f6b4aefc830f0f84b3f1d9e1 WatchSource:0}: Error finding container d1d099edbbf4d3fed1cfa850998fcbc526a836d6f6b4aefc830f0f84b3f1d9e1: Status 404 returned error can't find the container with id d1d099edbbf4d3fed1cfa850998fcbc526a836d6f6b4aefc830f0f84b3f1d9e1
Sep 11 16:40:44.625786 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:44.625750    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" event={"ID":"b98ca9c4-cabf-49a1-aa03-46191cde0bce","Type":"ContainerStarted","Data":"9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1"}
Sep 11 16:40:44.625967 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:44.625799    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" event={"ID":"b98ca9c4-cabf-49a1-aa03-46191cde0bce","Type":"ContainerStarted","Data":"d1d099edbbf4d3fed1cfa850998fcbc526a836d6f6b4aefc830f0f84b3f1d9e1"}
Sep 11 16:40:44.625967 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:44.625839    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:40:44.642072 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:44.642017    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podStartSLOduration=1.642003186 podStartE2EDuration="1.642003186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:40:44.641052675 +0000 UTC m=+1419.597100303" watchObservedRunningTime="2026-09-11 16:40:44.642003186 +0000 UTC m=+1419.598050815"
Sep 11 16:40:45.477458 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:40:45.477423    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14819bb2-341b-4321-b514-7d7df37a4491" path="/var/lib/kubelet/pods/14819bb2-341b-4321-b514-7d7df37a4491/volumes"
Sep 11 16:41:07.274765 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.274730    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"]
Sep 11 16:41:07.277847 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.277825    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.280304 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.280279    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-mt9br\""
Sep 11 16:41:07.280435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.280316    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 11 16:41:07.280435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.280330    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 11 16:41:07.280435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.280334    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 11 16:41:07.280435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.280359    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-lztl9\""
Sep 11 16:41:07.291403 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.288108    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"]
Sep 11 16:41:07.461275 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.461244    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74f7ff6-c5e7-4448-8072-ff2773460ad2-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.461413 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.461278    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svwhx\" (UniqueName: \"kubernetes.io/projected/a74f7ff6-c5e7-4448-8072-ff2773460ad2-kube-api-access-svwhx\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.562701 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.562632    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74f7ff6-c5e7-4448-8072-ff2773460ad2-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.562701 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.562665    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svwhx\" (UniqueName: \"kubernetes.io/projected/a74f7ff6-c5e7-4448-8072-ff2773460ad2-kube-api-access-svwhx\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.565145 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.565126    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74f7ff6-c5e7-4448-8072-ff2773460ad2-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.569189 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.569169    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svwhx\" (UniqueName: \"kubernetes.io/projected/a74f7ff6-c5e7-4448-8072-ff2773460ad2-kube-api-access-svwhx\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.590118 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.590098    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:07.861001 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:07.860979    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"]
Sep 11 16:41:07.863020 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:41:07.862996    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74f7ff6_c5e7_4448_8072_ff2773460ad2.slice/crio-b353894fab1a0e32a1315aa670f47f89b095fc6d0ccd584cebfd33356aa2c2c8 WatchSource:0}: Error finding container b353894fab1a0e32a1315aa670f47f89b095fc6d0ccd584cebfd33356aa2c2c8: Status 404 returned error can't find the container with id b353894fab1a0e32a1315aa670f47f89b095fc6d0ccd584cebfd33356aa2c2c8
Sep 11 16:41:08.740732 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:08.740697    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" event={"ID":"a74f7ff6-c5e7-4448-8072-ff2773460ad2","Type":"ContainerStarted","Data":"f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c"}
Sep 11 16:41:08.741112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:08.740756    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" event={"ID":"a74f7ff6-c5e7-4448-8072-ff2773460ad2","Type":"ContainerStarted","Data":"b353894fab1a0e32a1315aa670f47f89b095fc6d0ccd584cebfd33356aa2c2c8"}
Sep 11 16:41:08.741112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:08.740795    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:08.761383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:08.761309    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" podStartSLOduration=1.761283312 podStartE2EDuration="1.761283312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:41:08.756539392 +0000 UTC m=+1443.712587023" watchObservedRunningTime="2026-09-11 16:41:08.761283312 +0000 UTC m=+1443.717330942"
Sep 11 16:41:15.635129 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:15.635098    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:41:39.747820 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:39.747788    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:41:41.434847 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:41.434813    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"]
Sep 11 16:41:49.746893 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:49.746857    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:41:56.454482 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:56.454443    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main" containerID="cri-o://f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c" gracePeriod=45
Sep 11 16:41:59.747487 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:41:59.747452    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:42:05.524512 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:05.524479    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:42:05.528889 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:05.528869    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:42:09.747481 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:09.747438    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:42:09.747900 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:09.747582    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:42:19.747478 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:19.747438    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:42:26.699004 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.698974    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:42:26.819075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.818999    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74f7ff6-c5e7-4448-8072-ff2773460ad2-tls-certs\") pod \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") "
Sep 11 16:42:26.819075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.819048    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svwhx\" (UniqueName: \"kubernetes.io/projected/a74f7ff6-c5e7-4448-8072-ff2773460ad2-kube-api-access-svwhx\") pod \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\" (UID: \"a74f7ff6-c5e7-4448-8072-ff2773460ad2\") "
Sep 11 16:42:26.821109 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.821085    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74f7ff6-c5e7-4448-8072-ff2773460ad2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a74f7ff6-c5e7-4448-8072-ff2773460ad2" (UID: "a74f7ff6-c5e7-4448-8072-ff2773460ad2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:42:26.821219 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.821143    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74f7ff6-c5e7-4448-8072-ff2773460ad2-kube-api-access-svwhx" (OuterVolumeSpecName: "kube-api-access-svwhx") pod "a74f7ff6-c5e7-4448-8072-ff2773460ad2" (UID: "a74f7ff6-c5e7-4448-8072-ff2773460ad2"). InnerVolumeSpecName "kube-api-access-svwhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:42:26.919905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.919875    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a74f7ff6-c5e7-4448-8072-ff2773460ad2-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:42:26.919905 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:26.919904    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svwhx\" (UniqueName: \"kubernetes.io/projected/a74f7ff6-c5e7-4448-8072-ff2773460ad2-kube-api-access-svwhx\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:42:27.112859 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.112833    2550 generic.go:356] "Generic (PLEG): container finished" podID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerID="f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c" exitCode=0
Sep 11 16:42:27.112997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.112874    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"
Sep 11 16:42:27.112997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.112922    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" event={"ID":"a74f7ff6-c5e7-4448-8072-ff2773460ad2","Type":"ContainerDied","Data":"f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c"}
Sep 11 16:42:27.112997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.112980    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t" event={"ID":"a74f7ff6-c5e7-4448-8072-ff2773460ad2","Type":"ContainerDied","Data":"b353894fab1a0e32a1315aa670f47f89b095fc6d0ccd584cebfd33356aa2c2c8"}
Sep 11 16:42:27.113102 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.113007    2550 scope.go:117] "RemoveContainer" containerID="f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c"
Sep 11 16:42:27.122039 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.122019    2550 scope.go:117] "RemoveContainer" containerID="f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c"
Sep 11 16:42:27.122315 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:42:27.122294    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c\": container with ID starting with f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c not found: ID does not exist" containerID="f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c"
Sep 11 16:42:27.122441 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.122322    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c"} err="failed to get container status \"f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c\": rpc error: code = NotFound desc = could not find container \"f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c\": container with ID starting with f8ce2d2735f103475bbdf911f1e2771c6c27f94471a71a28bc29fcd8be7e227c not found: ID does not exist"
Sep 11 16:42:27.133065 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.133046    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"]
Sep 11 16:42:27.137009 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.136989    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche5k44t"]
Sep 11 16:42:27.477287 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:27.477212    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" path="/var/lib/kubelet/pods/a74f7ff6-c5e7-4448-8072-ff2773460ad2/volumes"
Sep 11 16:42:49.594811 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.594776    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"]
Sep 11 16:42:49.597046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.595095    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main"
Sep 11 16:42:49.597046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.595106    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main"
Sep 11 16:42:49.597046 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.595158    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74f7ff6-c5e7-4448-8072-ff2773460ad2" containerName="main"
Sep 11 16:42:49.597904 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.597888    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.600330 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.600297    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-lb5jw\""
Sep 11 16:42:49.600330 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.600312    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 11 16:42:49.600506 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.600347    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-t7llv\""
Sep 11 16:42:49.600506 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.600406    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 11 16:42:49.600506 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.600445    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 11 16:42:49.609372 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.609349    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"]
Sep 11 16:42:49.677320 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.677290    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss9hn\" (UniqueName: \"kubernetes.io/projected/19629120-b0b5-4364-bfd9-40c09a27e40d-kube-api-access-ss9hn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.677469 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.677370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19629120-b0b5-4364-bfd9-40c09a27e40d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.778239 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.778211    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19629120-b0b5-4364-bfd9-40c09a27e40d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.778400 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.778252    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss9hn\" (UniqueName: \"kubernetes.io/projected/19629120-b0b5-4364-bfd9-40c09a27e40d-kube-api-access-ss9hn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.780711 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.780687    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19629120-b0b5-4364-bfd9-40c09a27e40d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.785527 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.785507    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss9hn\" (UniqueName: \"kubernetes.io/projected/19629120-b0b5-4364-bfd9-40c09a27e40d-kube-api-access-ss9hn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:49.910345 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:49.910281    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:50.180277 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:50.180096    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"]
Sep 11 16:42:50.182476 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:42:50.182447    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19629120_b0b5_4364_bfd9_40c09a27e40d.slice/crio-230ff210f4dd1d539f6ff8aa22a8d3bb16d80f7ef97745676c50a4e9180fae3c WatchSource:0}: Error finding container 230ff210f4dd1d539f6ff8aa22a8d3bb16d80f7ef97745676c50a4e9180fae3c: Status 404 returned error can't find the container with id 230ff210f4dd1d539f6ff8aa22a8d3bb16d80f7ef97745676c50a4e9180fae3c
Sep 11 16:42:50.227601 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:50.227568    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" event={"ID":"19629120-b0b5-4364-bfd9-40c09a27e40d","Type":"ContainerStarted","Data":"230ff210f4dd1d539f6ff8aa22a8d3bb16d80f7ef97745676c50a4e9180fae3c"}
Sep 11 16:42:51.234213 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:51.234177    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" event={"ID":"19629120-b0b5-4364-bfd9-40c09a27e40d","Type":"ContainerStarted","Data":"9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860"}
Sep 11 16:42:51.234714 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:51.234262    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:42:51.251102 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:42:51.251052    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" podStartSLOduration=2.251037987 podStartE2EDuration="2.251037987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:42:51.248736241 +0000 UTC m=+1546.204783870" watchObservedRunningTime="2026-09-11 16:42:51.251037987 +0000 UTC m=+1546.207085631"
Sep 11 16:43:04.947361 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:04.947323    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"]
Sep 11 16:43:05.634793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:05.634762    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:15.634744 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:15.634707    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:19.967027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:19.966971    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main" containerID="cri-o://9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1" gracePeriod=45
Sep 11 16:43:22.241123 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:22.241094    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:43:23.917520 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:23.917484    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"]
Sep 11 16:43:25.635053 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:25.635018    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:25.635428 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:25.635105    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:43:32.241461 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:32.241419    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:35.634555 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:35.634500    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:38.938691 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:38.938650    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main" containerID="cri-o://9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860" gracePeriod=45
Sep 11 16:43:42.241042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:42.241007    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:45.634775 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:45.634737    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:50.215042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.215017    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:43:50.329656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.329632    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch2tb\" (UniqueName: \"kubernetes.io/projected/b98ca9c4-cabf-49a1-aa03-46191cde0bce-kube-api-access-ch2tb\") pod \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") "
Sep 11 16:43:50.329777 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.329673    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98ca9c4-cabf-49a1-aa03-46191cde0bce-tls-certs\") pod \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\" (UID: \"b98ca9c4-cabf-49a1-aa03-46191cde0bce\") "
Sep 11 16:43:50.331738 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.331708    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98ca9c4-cabf-49a1-aa03-46191cde0bce-kube-api-access-ch2tb" (OuterVolumeSpecName: "kube-api-access-ch2tb") pod "b98ca9c4-cabf-49a1-aa03-46191cde0bce" (UID: "b98ca9c4-cabf-49a1-aa03-46191cde0bce"). InnerVolumeSpecName "kube-api-access-ch2tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:50.331852 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.331771    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98ca9c4-cabf-49a1-aa03-46191cde0bce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b98ca9c4-cabf-49a1-aa03-46191cde0bce" (UID: "b98ca9c4-cabf-49a1-aa03-46191cde0bce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:43:50.430933 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.430905    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch2tb\" (UniqueName: \"kubernetes.io/projected/b98ca9c4-cabf-49a1-aa03-46191cde0bce-kube-api-access-ch2tb\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:43:50.431031 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.430931    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98ca9c4-cabf-49a1-aa03-46191cde0bce-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:43:50.514491 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.514468    2550 generic.go:356] "Generic (PLEG): container finished" podID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerID="9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1" exitCode=0
Sep 11 16:43:50.514603 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.514503    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" event={"ID":"b98ca9c4-cabf-49a1-aa03-46191cde0bce","Type":"ContainerDied","Data":"9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1"}
Sep 11 16:43:50.514603 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.514515    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"
Sep 11 16:43:50.514603 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.514536    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf" event={"ID":"b98ca9c4-cabf-49a1-aa03-46191cde0bce","Type":"ContainerDied","Data":"d1d099edbbf4d3fed1cfa850998fcbc526a836d6f6b4aefc830f0f84b3f1d9e1"}
Sep 11 16:43:50.514603 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.514572    2550 scope.go:117] "RemoveContainer" containerID="9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1"
Sep 11 16:43:50.523483 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.523465    2550 scope.go:117] "RemoveContainer" containerID="9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1"
Sep 11 16:43:50.523752 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:43:50.523732    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1\": container with ID starting with 9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1 not found: ID does not exist" containerID="9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1"
Sep 11 16:43:50.523804 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.523762    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1"} err="failed to get container status \"9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1\": rpc error: code = NotFound desc = could not find container \"9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1\": container with ID starting with 9a500c8aff46e2a2118bfed76770bb33363151a1e0ab1255472406fa13d98da1 not found: ID does not exist"
Sep 11 16:43:50.534916 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.534897    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"]
Sep 11 16:43:50.538899 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:50.538875    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cmzbmf"]
Sep 11 16:43:51.476677 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:51.476642    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" path="/var/lib/kubelet/pods/b98ca9c4-cabf-49a1-aa03-46191cde0bce/volumes"
Sep 11 16:43:52.240796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:52.240763    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:43:52.240982 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:43:52.240854    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:44:02.241868 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:02.241777    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:44:09.183866 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.183840    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:44:09.269694 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.269627    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss9hn\" (UniqueName: \"kubernetes.io/projected/19629120-b0b5-4364-bfd9-40c09a27e40d-kube-api-access-ss9hn\") pod \"19629120-b0b5-4364-bfd9-40c09a27e40d\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") "
Sep 11 16:44:09.269827 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.269752    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19629120-b0b5-4364-bfd9-40c09a27e40d-tls-certs\") pod \"19629120-b0b5-4364-bfd9-40c09a27e40d\" (UID: \"19629120-b0b5-4364-bfd9-40c09a27e40d\") "
Sep 11 16:44:09.271862 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.271833    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19629120-b0b5-4364-bfd9-40c09a27e40d-kube-api-access-ss9hn" (OuterVolumeSpecName: "kube-api-access-ss9hn") pod "19629120-b0b5-4364-bfd9-40c09a27e40d" (UID: "19629120-b0b5-4364-bfd9-40c09a27e40d"). InnerVolumeSpecName "kube-api-access-ss9hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:44:09.271862 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.271856    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19629120-b0b5-4364-bfd9-40c09a27e40d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19629120-b0b5-4364-bfd9-40c09a27e40d" (UID: "19629120-b0b5-4364-bfd9-40c09a27e40d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:44:09.370374 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.370344    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19629120-b0b5-4364-bfd9-40c09a27e40d-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:44:09.370374 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.370370    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss9hn\" (UniqueName: \"kubernetes.io/projected/19629120-b0b5-4364-bfd9-40c09a27e40d-kube-api-access-ss9hn\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:44:09.603785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.603758    2550 generic.go:356] "Generic (PLEG): container finished" podID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerID="9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860" exitCode=0
Sep 11 16:44:09.603916 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.603800    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"
Sep 11 16:44:09.603916 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.603847    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" event={"ID":"19629120-b0b5-4364-bfd9-40c09a27e40d","Type":"ContainerDied","Data":"9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860"}
Sep 11 16:44:09.603916 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.603900    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv" event={"ID":"19629120-b0b5-4364-bfd9-40c09a27e40d","Type":"ContainerDied","Data":"230ff210f4dd1d539f6ff8aa22a8d3bb16d80f7ef97745676c50a4e9180fae3c"}
Sep 11 16:44:09.604080 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.603930    2550 scope.go:117] "RemoveContainer" containerID="9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860"
Sep 11 16:44:09.612443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.612421    2550 scope.go:117] "RemoveContainer" containerID="9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860"
Sep 11 16:44:09.612720 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:44:09.612699    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860\": container with ID starting with 9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860 not found: ID does not exist" containerID="9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860"
Sep 11 16:44:09.612778 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.612729    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860"} err="failed to get container status \"9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860\": rpc error: code = NotFound desc = could not find container \"9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860\": container with ID starting with 9320b16f9b70a010472e779038fdea3ea845abfea61ee78b4fdd38ed94056860 not found: ID does not exist"
Sep 11 16:44:09.618023 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.618003    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"]
Sep 11 16:44:09.622134 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:09.622116    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9dpddv"]
Sep 11 16:44:11.477970 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:11.477935    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" path="/var/lib/kubelet/pods/19629120-b0b5-4364-bfd9-40c09a27e40d/volumes"
Sep 11 16:44:20.391519 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391483    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"]
Sep 11 16:44:20.392042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391806    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main"
Sep 11 16:44:20.392042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391818    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main"
Sep 11 16:44:20.392042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391832    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main"
Sep 11 16:44:20.392042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391838    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main"
Sep 11 16:44:20.392042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391902    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="19629120-b0b5-4364-bfd9-40c09a27e40d" containerName="main"
Sep 11 16:44:20.392042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.391914    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98ca9c4-cabf-49a1-aa03-46191cde0bce" containerName="main"
Sep 11 16:44:20.402621 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.402587    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.404835 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.404807    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 11 16:44:20.404967 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.404874    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"]
Sep 11 16:44:20.405205 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.405182    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-7wlxl\""
Sep 11 16:44:20.405310 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.405185    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 11 16:44:20.405380 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.405312    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 11 16:44:20.405380 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.405262    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-457zj\""
Sep 11 16:44:20.540030 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.540002    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5217ec0-9b9d-438b-a1df-336421f74442-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.540181 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.540034    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42c2d\" (UniqueName: \"kubernetes.io/projected/e5217ec0-9b9d-438b-a1df-336421f74442-kube-api-access-42c2d\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.640624 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.640589    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5217ec0-9b9d-438b-a1df-336421f74442-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.640624 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.640630    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42c2d\" (UniqueName: \"kubernetes.io/projected/e5217ec0-9b9d-438b-a1df-336421f74442-kube-api-access-42c2d\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.643083 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.643029    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5217ec0-9b9d-438b-a1df-336421f74442-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.648937 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.648912    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42c2d\" (UniqueName: \"kubernetes.io/projected/e5217ec0-9b9d-438b-a1df-336421f74442-kube-api-access-42c2d\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.715182 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.715156    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:20.988128 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:20.988100    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"]
Sep 11 16:44:20.990002 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:44:20.989973    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5217ec0_9b9d_438b_a1df_336421f74442.slice/crio-ef80eda740eb5ff368148e6e89f0942120dfbb6ea33a29d6d7fe4a25ea2edb95 WatchSource:0}: Error finding container ef80eda740eb5ff368148e6e89f0942120dfbb6ea33a29d6d7fe4a25ea2edb95: Status 404 returned error can't find the container with id ef80eda740eb5ff368148e6e89f0942120dfbb6ea33a29d6d7fe4a25ea2edb95
Sep 11 16:44:21.662087 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:21.662049    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" event={"ID":"e5217ec0-9b9d-438b-a1df-336421f74442","Type":"ContainerStarted","Data":"41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add"}
Sep 11 16:44:21.662500 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:21.662107    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" event={"ID":"e5217ec0-9b9d-438b-a1df-336421f74442","Type":"ContainerStarted","Data":"ef80eda740eb5ff368148e6e89f0942120dfbb6ea33a29d6d7fe4a25ea2edb95"}
Sep 11 16:44:21.662500 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:21.662130    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:44:21.678288 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:21.678243    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podStartSLOduration=1.678205001 podStartE2EDuration="1.678205001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:21.676153484 +0000 UTC m=+1636.632201135" watchObservedRunningTime="2026-09-11 16:44:21.678205001 +0000 UTC m=+1636.634252628"
Sep 11 16:44:52.669879 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:44:52.669849    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:46:37.659994 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:46:37.659958    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"]
Sep 11 16:46:42.670258 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:46:42.670221    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:46:52.669957 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:46:52.669921    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:46:52.678599 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:46:52.678569    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main" containerID="cri-o://41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add" gracePeriod=45
Sep 11 16:47:02.670089 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:02.670003    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:47:02.670530 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:02.670116    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:47:05.553966 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:05.553938    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:47:05.560077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:05.560054    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:47:12.670020 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:12.669976    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:47:22.669714 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:22.669677    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:47:22.926276 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:22.926216    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:47:23.042658 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.042624    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5217ec0-9b9d-438b-a1df-336421f74442-tls-certs\") pod \"e5217ec0-9b9d-438b-a1df-336421f74442\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") "
Sep 11 16:47:23.042819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.042703    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42c2d\" (UniqueName: \"kubernetes.io/projected/e5217ec0-9b9d-438b-a1df-336421f74442-kube-api-access-42c2d\") pod \"e5217ec0-9b9d-438b-a1df-336421f74442\" (UID: \"e5217ec0-9b9d-438b-a1df-336421f74442\") "
Sep 11 16:47:23.044889 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.044862    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5217ec0-9b9d-438b-a1df-336421f74442-kube-api-access-42c2d" (OuterVolumeSpecName: "kube-api-access-42c2d") pod "e5217ec0-9b9d-438b-a1df-336421f74442" (UID: "e5217ec0-9b9d-438b-a1df-336421f74442"). InnerVolumeSpecName "kube-api-access-42c2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:47:23.044999 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.044882    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5217ec0-9b9d-438b-a1df-336421f74442-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5217ec0-9b9d-438b-a1df-336421f74442" (UID: "e5217ec0-9b9d-438b-a1df-336421f74442"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:47:23.143372 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.143351    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42c2d\" (UniqueName: \"kubernetes.io/projected/e5217ec0-9b9d-438b-a1df-336421f74442-kube-api-access-42c2d\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:47:23.143372 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.143371    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5217ec0-9b9d-438b-a1df-336421f74442-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:47:23.520594 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.520562    2550 generic.go:356] "Generic (PLEG): container finished" podID="e5217ec0-9b9d-438b-a1df-336421f74442" containerID="41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add" exitCode=0
Sep 11 16:47:23.520594 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.520597    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"
Sep 11 16:47:23.520813 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.520632    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" event={"ID":"e5217ec0-9b9d-438b-a1df-336421f74442","Type":"ContainerDied","Data":"41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add"}
Sep 11 16:47:23.520813 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.520693    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x" event={"ID":"e5217ec0-9b9d-438b-a1df-336421f74442","Type":"ContainerDied","Data":"ef80eda740eb5ff368148e6e89f0942120dfbb6ea33a29d6d7fe4a25ea2edb95"}
Sep 11 16:47:23.520813 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.520724    2550 scope.go:117] "RemoveContainer" containerID="41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add"
Sep 11 16:47:23.531512 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.531488    2550 scope.go:117] "RemoveContainer" containerID="41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add"
Sep 11 16:47:23.531805 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:47:23.531785    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add\": container with ID starting with 41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add not found: ID does not exist" containerID="41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add"
Sep 11 16:47:23.531876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.531812    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add"} err="failed to get container status \"41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add\": rpc error: code = NotFound desc = could not find container \"41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add\": container with ID starting with 41f89d856008a88b847850424b0d38c3c2b4c53d0c2e4b4e1954bccb82fa0add not found: ID does not exist"
Sep 11 16:47:23.535445 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.535390    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"]
Sep 11 16:47:23.537273 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:23.537255    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-wns2x"]
Sep 11 16:47:25.476556 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:25.476507    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" path="/var/lib/kubelet/pods/e5217ec0-9b9d-438b-a1df-336421f74442/volumes"
Sep 11 16:47:29.895260 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.895226    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"]
Sep 11 16:47:29.895658 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.895557    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main"
Sep 11 16:47:29.895658 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.895569    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main"
Sep 11 16:47:29.895658 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.895630    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5217ec0-9b9d-438b-a1df-336421f74442" containerName="main"
Sep 11 16:47:29.899925 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.899902    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:29.904335 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.904315    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 11 16:47:29.904697 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.904679    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 11 16:47:29.904796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.904724    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-ms8f7\""
Sep 11 16:47:29.904796 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.904774    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 11 16:47:29.912239 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.912210    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"]
Sep 11 16:47:29.995610 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.995584    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784654f0-fc59-4278-9e38-d7d8e7936a32-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:29.995719 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.995616    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vht7d\" (UniqueName: \"kubernetes.io/projected/784654f0-fc59-4278-9e38-d7d8e7936a32-kube-api-access-vht7d\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:29.995719 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.995652    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-model-cache\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:29.995798 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.995722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-dshm\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:29.995798 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.995771    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-home\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:29.995863 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:29.995814    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096144 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096115    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-model-cache\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096153    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-dshm\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096175    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-home\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096215    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096243    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784654f0-fc59-4278-9e38-d7d8e7936a32-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096445 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096272    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht7d\" (UniqueName: \"kubernetes.io/projected/784654f0-fc59-4278-9e38-d7d8e7936a32-kube-api-access-vht7d\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096668 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096648    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-home\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096743 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096704    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-model-cache\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.096743 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.096728    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.098495 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.098474    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-dshm\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.098719 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.098700    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784654f0-fc59-4278-9e38-d7d8e7936a32-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.103651 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.103630    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht7d\" (UniqueName: \"kubernetes.io/projected/784654f0-fc59-4278-9e38-d7d8e7936a32-kube-api-access-vht7d\") pod \"canary-v1-kserve-65d58ccc9-7zxjn\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.210363 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.210300    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:30.481277 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.481254    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"]
Sep 11 16:47:30.483230 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:47:30.483207    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784654f0_fc59_4278_9e38_d7d8e7936a32.slice/crio-c485c4edcffc86c6daaa836b5709bf4ce6890a8cf4ee38bb17595e3849e8f093 WatchSource:0}: Error finding container c485c4edcffc86c6daaa836b5709bf4ce6890a8cf4ee38bb17595e3849e8f093: Status 404 returned error can't find the container with id c485c4edcffc86c6daaa836b5709bf4ce6890a8cf4ee38bb17595e3849e8f093
Sep 11 16:47:30.485357 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.485338    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:47:30.561133 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:30.561099    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn" event={"ID":"784654f0-fc59-4278-9e38-d7d8e7936a32","Type":"ContainerStarted","Data":"c485c4edcffc86c6daaa836b5709bf4ce6890a8cf4ee38bb17595e3849e8f093"}
Sep 11 16:47:32.573156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:32.573120    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn" event={"ID":"784654f0-fc59-4278-9e38-d7d8e7936a32","Type":"ContainerStarted","Data":"f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92"}
Sep 11 16:47:32.587983 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:32.587938    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn" podStartSLOduration=2.4364348 podStartE2EDuration="3.587920172s" totalImagesPullingTime="1.151485372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:32.587333774 +0000 UTC m=+1827.543381403" watchObservedRunningTime="2026-09-11 16:47:32.587920172 +0000 UTC m=+1827.543967802"
Sep 11 16:47:33.211476 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:33.211444    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:40.211078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:40.211044    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:40.223714 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:40.223690    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:40.621398 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:40.621370    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:47:49.681765 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.681656    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"]
Sep 11 16:47:49.686574 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.686530    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.688529 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.688506    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 11 16:47:49.692310 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.692286    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"]
Sep 11 16:47:49.740885 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.740855    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-home\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.741034 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.740893    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-dshm\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.741034 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.740912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-845dg\" (UniqueName: \"kubernetes.io/projected/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-kube-api-access-845dg\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.741034 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.740978    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.741034 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.741021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.741180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.741082    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842033 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842001    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842172 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842118    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-home\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842172 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842155    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-dshm\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842176    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-845dg\" (UniqueName: \"kubernetes.io/projected/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-kube-api-access-845dg\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842212    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842236    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842417 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842343    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842515 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842493    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-home\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.842680 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.842657    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.844476 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.844448    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-dshm\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.844793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.844773    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.849538 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.849518    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-845dg\" (UniqueName: \"kubernetes.io/projected/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-kube-api-access-845dg\") pod \"canary-v2-kserve-7f65dbbccc-z2z97\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:49.998661 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:49.998578    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:50.272751 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:50.272727    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"]
Sep 11 16:47:50.274692 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:47:50.274662    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3911ceb9_a2e4_4f77_b5ba_fcc5ee73b0af.slice/crio-c0155f914acae6b726e396fb12be562a765e629f07c3afffca67b69e5bd19029 WatchSource:0}: Error finding container c0155f914acae6b726e396fb12be562a765e629f07c3afffca67b69e5bd19029: Status 404 returned error can't find the container with id c0155f914acae6b726e396fb12be562a765e629f07c3afffca67b69e5bd19029
Sep 11 16:47:50.658554 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:50.658514    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" event={"ID":"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af","Type":"ContainerStarted","Data":"d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61"}
Sep 11 16:47:50.658721 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:50.658587    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" event={"ID":"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af","Type":"ContainerStarted","Data":"c0155f914acae6b726e396fb12be562a765e629f07c3afffca67b69e5bd19029"}
Sep 11 16:47:50.674466 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:50.674419    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" podStartSLOduration=1.67440168 podStartE2EDuration="1.67440168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:47:50.672941134 +0000 UTC m=+1845.628988762" watchObservedRunningTime="2026-09-11 16:47:50.67440168 +0000 UTC m=+1845.630449307"
Sep 11 16:47:50.999045 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:50.998970    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:47:59.998867 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:47:59.998826    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:48:00.011507 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:00.011478    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:48:00.717384 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:00.717359    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:48:08.025012 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.024980    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"]
Sep 11 16:48:08.028435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.028413    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.030449 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.030425    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 11 16:48:08.030782 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.030760    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6vttl\""
Sep 11 16:48:08.030881 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.030811    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 11 16:48:08.030881 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.030818    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 11 16:48:08.045400 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.045375    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"]
Sep 11 16:48:08.065141 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.065108    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.065275 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.065212    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.065275 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.065239    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrld\" (UniqueName: \"kubernetes.io/projected/ef648d29-870f-4bd8-bad2-b7d1445ccf63-kube-api-access-lhrld\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.065400 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.065272    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.065400 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.065305    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.065400 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.065334    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.165879 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.165848    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.165879 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.165880    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrld\" (UniqueName: \"kubernetes.io/projected/ef648d29-870f-4bd8-bad2-b7d1445ccf63-kube-api-access-lhrld\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.165902    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.165923    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.165940    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.165972    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.166293    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166374 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.166353    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.166410 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.166380    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.168175 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.168153    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.168420 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.168405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.172942 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.172917    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrld\" (UniqueName: \"kubernetes.io/projected/ef648d29-870f-4bd8-bad2-b7d1445ccf63-kube-api-access-lhrld\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.344834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.344803    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:08.613498 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.613474    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"]
Sep 11 16:48:08.615512 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:48:08.615485    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef648d29_870f_4bd8_bad2_b7d1445ccf63.slice/crio-692003ea0e2a7c4011bc1abaf75bb79aa8df362e099cec6ce944473bec08e109 WatchSource:0}: Error finding container 692003ea0e2a7c4011bc1abaf75bb79aa8df362e099cec6ce944473bec08e109: Status 404 returned error can't find the container with id 692003ea0e2a7c4011bc1abaf75bb79aa8df362e099cec6ce944473bec08e109
Sep 11 16:48:08.744989 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.744953    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" event={"ID":"ef648d29-870f-4bd8-bad2-b7d1445ccf63","Type":"ContainerStarted","Data":"ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"}
Sep 11 16:48:08.745133 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.745003    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" event={"ID":"ef648d29-870f-4bd8-bad2-b7d1445ccf63","Type":"ContainerStarted","Data":"692003ea0e2a7c4011bc1abaf75bb79aa8df362e099cec6ce944473bec08e109"}
Sep 11 16:48:08.762760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:08.762710    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" podStartSLOduration=0.76269698 podStartE2EDuration="762.69698ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:08.760301397 +0000 UTC m=+1863.716349025" watchObservedRunningTime="2026-09-11 16:48:08.76269698 +0000 UTC m=+1863.718744608"
Sep 11 16:48:09.345054 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:09.345022    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:18.345279 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:18.345236    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:18.357840 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:18.357812    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:48:18.802022 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:48:18.801942    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:49:38.760407 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:38.760374    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"]
Sep 11 16:49:39.480424 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:39.480393    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"]
Sep 11 16:49:53.782042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:53.781996    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn" podUID="784654f0-fc59-4278-9e38-d7d8e7936a32" containerName="main" containerID="cri-o://f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92" gracePeriod=45
Sep 11 16:49:54.022267 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.022242    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:49:54.094388 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094360    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-home\") pod \"784654f0-fc59-4278-9e38-d7d8e7936a32\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") "
Sep 11 16:49:54.094522 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094396    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vht7d\" (UniqueName: \"kubernetes.io/projected/784654f0-fc59-4278-9e38-d7d8e7936a32-kube-api-access-vht7d\") pod \"784654f0-fc59-4278-9e38-d7d8e7936a32\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") "
Sep 11 16:49:54.094522 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094420    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-model-cache\") pod \"784654f0-fc59-4278-9e38-d7d8e7936a32\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") "
Sep 11 16:49:54.094522 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094453    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784654f0-fc59-4278-9e38-d7d8e7936a32-tls-certs\") pod \"784654f0-fc59-4278-9e38-d7d8e7936a32\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") "
Sep 11 16:49:54.094702 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094579    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-dshm\") pod \"784654f0-fc59-4278-9e38-d7d8e7936a32\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") "
Sep 11 16:49:54.094702 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094657    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-tmp-dir\") pod \"784654f0-fc59-4278-9e38-d7d8e7936a32\" (UID: \"784654f0-fc59-4278-9e38-d7d8e7936a32\") "
Sep 11 16:49:54.094702 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094664    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-home" (OuterVolumeSpecName: "home") pod "784654f0-fc59-4278-9e38-d7d8e7936a32" (UID: "784654f0-fc59-4278-9e38-d7d8e7936a32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.094882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094779    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-model-cache" (OuterVolumeSpecName: "model-cache") pod "784654f0-fc59-4278-9e38-d7d8e7936a32" (UID: "784654f0-fc59-4278-9e38-d7d8e7936a32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.095027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.094992    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "784654f0-fc59-4278-9e38-d7d8e7936a32" (UID: "784654f0-fc59-4278-9e38-d7d8e7936a32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.095147 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.095004    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.095147 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.095055    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.096705 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.096684    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784654f0-fc59-4278-9e38-d7d8e7936a32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "784654f0-fc59-4278-9e38-d7d8e7936a32" (UID: "784654f0-fc59-4278-9e38-d7d8e7936a32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:49:54.096784 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.096703    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784654f0-fc59-4278-9e38-d7d8e7936a32-kube-api-access-vht7d" (OuterVolumeSpecName: "kube-api-access-vht7d") pod "784654f0-fc59-4278-9e38-d7d8e7936a32" (UID: "784654f0-fc59-4278-9e38-d7d8e7936a32"). InnerVolumeSpecName "kube-api-access-vht7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:49:54.096784 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.096742    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-dshm" (OuterVolumeSpecName: "dshm") pod "784654f0-fc59-4278-9e38-d7d8e7936a32" (UID: "784654f0-fc59-4278-9e38-d7d8e7936a32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.196090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.196065    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vht7d\" (UniqueName: \"kubernetes.io/projected/784654f0-fc59-4278-9e38-d7d8e7936a32-kube-api-access-vht7d\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.196090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.196088    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784654f0-fc59-4278-9e38-d7d8e7936a32-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.196237 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.196099    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.196237 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.196106    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784654f0-fc59-4278-9e38-d7d8e7936a32-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.256578 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.256534    2550 generic.go:356] "Generic (PLEG): container finished" podID="784654f0-fc59-4278-9e38-d7d8e7936a32" containerID="f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92" exitCode=0
Sep 11 16:49:54.256680 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.256602    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"
Sep 11 16:49:54.256680 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.256627    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn" event={"ID":"784654f0-fc59-4278-9e38-d7d8e7936a32","Type":"ContainerDied","Data":"f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92"}
Sep 11 16:49:54.256680 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.256678    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn" event={"ID":"784654f0-fc59-4278-9e38-d7d8e7936a32","Type":"ContainerDied","Data":"c485c4edcffc86c6daaa836b5709bf4ce6890a8cf4ee38bb17595e3849e8f093"}
Sep 11 16:49:54.256802 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.256699    2550 scope.go:117] "RemoveContainer" containerID="f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92"
Sep 11 16:49:54.265795 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.265777    2550 scope.go:117] "RemoveContainer" containerID="f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92"
Sep 11 16:49:54.266068 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:49:54.266050    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92\": container with ID starting with f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92 not found: ID does not exist" containerID="f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92"
Sep 11 16:49:54.266128 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.266077    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92"} err="failed to get container status \"f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92\": rpc error: code = NotFound desc = could not find container \"f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92\": container with ID starting with f435bce65ab18896fc450aef3e80cf771da647327e7648d4cfe3be55eb285b92 not found: ID does not exist"
Sep 11 16:49:54.277202 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.277182    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"]
Sep 11 16:49:54.280760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.280740    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-7zxjn"]
Sep 11 16:49:54.499112 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.499011    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" podUID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" containerName="main" containerID="cri-o://d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61" gracePeriod=45
Sep 11 16:49:54.730247 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.730223    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:49:54.800986 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.800958    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-model-cache\") pod \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") "
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.800989    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-dshm\") pod \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") "
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801055    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-home\") pod \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") "
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801077    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tls-certs\") pod \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") "
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801098    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-845dg\" (UniqueName: \"kubernetes.io/projected/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-kube-api-access-845dg\") pod \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") "
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801148    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tmp-dir\") pod \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\" (UID: \"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af\") "
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801256    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-model-cache" (OuterVolumeSpecName: "model-cache") pod "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" (UID: "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801325    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-home" (OuterVolumeSpecName: "home") pod "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" (UID: "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.801365 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801365    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.801818 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.801459    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" (UID: "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.803226 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.803197    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-kube-api-access-845dg" (OuterVolumeSpecName: "kube-api-access-845dg") pod "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" (UID: "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af"). InnerVolumeSpecName "kube-api-access-845dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:49:54.803334 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.803228    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-dshm" (OuterVolumeSpecName: "dshm") pod "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" (UID: "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:49:54.803334 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.803237    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" (UID: "3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:49:54.901840 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.901822    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.901840 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.901840    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.901962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.901849    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-845dg\" (UniqueName: \"kubernetes.io/projected/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-kube-api-access-845dg\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.901962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.901857    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:54.901962 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:54.901866    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:49:55.264304 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.264269    2550 generic.go:356] "Generic (PLEG): container finished" podID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" containerID="d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61" exitCode=0
Sep 11 16:49:55.264458 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.264317    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"
Sep 11 16:49:55.264458 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.264340    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" event={"ID":"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af","Type":"ContainerDied","Data":"d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61"}
Sep 11 16:49:55.264458 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.264392    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" event={"ID":"3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af","Type":"ContainerDied","Data":"c0155f914acae6b726e396fb12be562a765e629f07c3afffca67b69e5bd19029"}
Sep 11 16:49:55.264458 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.264425    2550 scope.go:117] "RemoveContainer" containerID="d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61"
Sep 11 16:49:55.273132 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.273116    2550 scope.go:117] "RemoveContainer" containerID="d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61"
Sep 11 16:49:55.273407 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:49:55.273387    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61\": container with ID starting with d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61 not found: ID does not exist" containerID="d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61"
Sep 11 16:49:55.273528 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.273414    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61"} err="failed to get container status \"d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61\": rpc error: code = NotFound desc = could not find container \"d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61\": container with ID starting with d37a3cb39b8ae455d5b2d5d5fb36035d5aac65603d4877d9ec04a8bf28911d61 not found: ID does not exist"
Sep 11 16:49:55.299900 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.299877    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"]
Sep 11 16:49:55.306841 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.306821    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97"]
Sep 11 16:49:55.478725 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.478692    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" path="/var/lib/kubelet/pods/3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af/volumes"
Sep 11 16:49:55.479078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.479065    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784654f0-fc59-4278-9e38-d7d8e7936a32" path="/var/lib/kubelet/pods/784654f0-fc59-4278-9e38-d7d8e7936a32/volumes"
Sep 11 16:49:55.707955 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:49:55.707911    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-z2z97" podUID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Sep 11 16:50:00.188487 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.188401    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"]
Sep 11 16:50:00.188997 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.188976    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784654f0-fc59-4278-9e38-d7d8e7936a32" containerName="main"
Sep 11 16:50:00.189075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.189000    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="784654f0-fc59-4278-9e38-d7d8e7936a32" containerName="main"
Sep 11 16:50:00.189075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.189011    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" containerName="main"
Sep 11 16:50:00.189075 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.189019    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" containerName="main"
Sep 11 16:50:00.189230 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.189111    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="784654f0-fc59-4278-9e38-d7d8e7936a32" containerName="main"
Sep 11 16:50:00.189230 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.189124    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3911ceb9-a2e4-4f77-b5ba-fcc5ee73b0af" containerName="main"
Sep 11 16:50:00.193506 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.193477    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.195297 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.195278    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 11 16:50:00.195428 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.195342    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 11 16:50:00.195797 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.195781    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 11 16:50:00.195855 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.195783    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-bcw6h\""
Sep 11 16:50:00.201794 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.201771    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"]
Sep 11 16:50:00.345855 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.345804    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.346005 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.345866    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-dshm\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.346005 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.345932    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-model-cache\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.346005 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.345992    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7qv\" (UniqueName: \"kubernetes.io/projected/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-kube-api-access-jx7qv\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.346136 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.346026    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-home\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.346136 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.346045    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447041 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.446965    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7qv\" (UniqueName: \"kubernetes.io/projected/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-kube-api-access-jx7qv\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447041 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447006    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-home\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447041 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447032    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447287 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447068    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447287 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447112    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-dshm\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447287 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447142    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-model-cache\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-home\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447419    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.447582 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.447514    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-model-cache\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.449350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.449329    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-dshm\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.449652 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.449631    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tls-certs\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.454084 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.454062    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7qv\" (UniqueName: \"kubernetes.io/projected/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-kube-api-access-jx7qv\") pod \"test-div-v1-kserve-fb566d667-xg62b\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.504984 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.504956    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:00.777519 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:00.777494    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"]
Sep 11 16:50:00.779375 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:50:00.779339    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862f9a49_2be2_4ef7_bb9e_7dd7c235bfc8.slice/crio-584bb06807dc7583da6a4e6072b5cf64f17c7138201f7372bc99dddd1a5722bc WatchSource:0}: Error finding container 584bb06807dc7583da6a4e6072b5cf64f17c7138201f7372bc99dddd1a5722bc: Status 404 returned error can't find the container with id 584bb06807dc7583da6a4e6072b5cf64f17c7138201f7372bc99dddd1a5722bc
Sep 11 16:50:01.294254 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:01.294221    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" event={"ID":"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8","Type":"ContainerStarted","Data":"b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d"}
Sep 11 16:50:01.294677 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:01.294279    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" event={"ID":"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8","Type":"ContainerStarted","Data":"584bb06807dc7583da6a4e6072b5cf64f17c7138201f7372bc99dddd1a5722bc"}
Sep 11 16:50:01.311947 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:01.311899    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" podStartSLOduration=1.3118837 podStartE2EDuration="1.3118837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:50:01.3093862 +0000 UTC m=+1976.265433827" watchObservedRunningTime="2026-09-11 16:50:01.3118837 +0000 UTC m=+1976.267931328"
Sep 11 16:50:01.505386 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:01.505357    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:10.505174 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:10.505140    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:10.517555 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:10.517524    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:11.353342 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:11.353315    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:50:25.807691 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:25.807659    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"]
Sep 11 16:50:25.816608 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:50:25.816567    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:50:25.816608 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:50:25.816608 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:50:25.816608 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b" cmd=["/bin/sleep","15"]
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:25.816625    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" message="" err=<
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:50:25.816637    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" podUID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" containerName="main" containerID="cri-o://ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"
Sep 11 16:50:25.816909 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:25.816670    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" podUID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" containerName="main" containerID="cri-o://ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b" gracePeriod=60
Sep 11 16:50:26.158672 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.158647    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:50:26.248486 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248457    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-model-cache\") pod \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") "
Sep 11 16:50:26.248656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248524    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tmp-dir\") pod \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") "
Sep 11 16:50:26.248656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248568    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-home\") pod \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") "
Sep 11 16:50:26.248656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248592    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-dshm\") pod \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") "
Sep 11 16:50:26.248656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248625    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhrld\" (UniqueName: \"kubernetes.io/projected/ef648d29-870f-4bd8-bad2-b7d1445ccf63-kube-api-access-lhrld\") pod \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") "
Sep 11 16:50:26.248656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248658    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tls-certs\") pod \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\" (UID: \"ef648d29-870f-4bd8-bad2-b7d1445ccf63\") "
Sep 11 16:50:26.248929 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248792    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef648d29-870f-4bd8-bad2-b7d1445ccf63" (UID: "ef648d29-870f-4bd8-bad2-b7d1445ccf63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.248929 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248819    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef648d29-870f-4bd8-bad2-b7d1445ccf63" (UID: "ef648d29-870f-4bd8-bad2-b7d1445ccf63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.248929 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.248842    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-home" (OuterVolumeSpecName: "home") pod "ef648d29-870f-4bd8-bad2-b7d1445ccf63" (UID: "ef648d29-870f-4bd8-bad2-b7d1445ccf63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.250717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.250691    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef648d29-870f-4bd8-bad2-b7d1445ccf63" (UID: "ef648d29-870f-4bd8-bad2-b7d1445ccf63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:50:26.250787 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.250760    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef648d29-870f-4bd8-bad2-b7d1445ccf63-kube-api-access-lhrld" (OuterVolumeSpecName: "kube-api-access-lhrld") pod "ef648d29-870f-4bd8-bad2-b7d1445ccf63" (UID: "ef648d29-870f-4bd8-bad2-b7d1445ccf63"). InnerVolumeSpecName "kube-api-access-lhrld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:50:26.250787 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.250774    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-dshm" (OuterVolumeSpecName: "dshm") pod "ef648d29-870f-4bd8-bad2-b7d1445ccf63" (UID: "ef648d29-870f-4bd8-bad2-b7d1445ccf63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:50:26.349451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.349389    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.349411    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.349419    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.349427    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrld\" (UniqueName: \"kubernetes.io/projected/ef648d29-870f-4bd8-bad2-b7d1445ccf63-kube-api-access-lhrld\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.349437    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef648d29-870f-4bd8-bad2-b7d1445ccf63-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.349451 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.349445    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef648d29-870f-4bd8-bad2-b7d1445ccf63-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:50:26.416833 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.416803    2550 generic.go:356] "Generic (PLEG): container finished" podID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" containerID="ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b" exitCode=0
Sep 11 16:50:26.416950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.416853    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"
Sep 11 16:50:26.416950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.416888    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" event={"ID":"ef648d29-870f-4bd8-bad2-b7d1445ccf63","Type":"ContainerDied","Data":"ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"}
Sep 11 16:50:26.416950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.416938    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4" event={"ID":"ef648d29-870f-4bd8-bad2-b7d1445ccf63","Type":"ContainerDied","Data":"692003ea0e2a7c4011bc1abaf75bb79aa8df362e099cec6ce944473bec08e109"}
Sep 11 16:50:26.417059 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.416962    2550 scope.go:117] "RemoveContainer" containerID="ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"
Sep 11 16:50:26.425971 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.425952    2550 scope.go:117] "RemoveContainer" containerID="ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"
Sep 11 16:50:26.426256 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:50:26.426237    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b\": container with ID starting with ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b not found: ID does not exist" containerID="ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"
Sep 11 16:50:26.426305 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.426265    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b"} err="failed to get container status \"ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b\": rpc error: code = NotFound desc = could not find container \"ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b\": container with ID starting with ebfd5671a31f400580761a177417260e422cbb7678e490cd700a7c7069c3be2b not found: ID does not exist"
Sep 11 16:50:26.439273 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.439249    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"]
Sep 11 16:50:26.445550 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:26.445521    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-9pcq4"]
Sep 11 16:50:27.477097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:27.477062    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" path="/var/lib/kubelet/pods/ef648d29-870f-4bd8-bad2-b7d1445ccf63/volumes"
Sep 11 16:50:45.280422 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:45.280385    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"]
Sep 11 16:50:58.986427 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.986395    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"]
Sep 11 16:50:58.986890 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.986755    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" containerName="main"
Sep 11 16:50:58.986890 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.986767    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" containerName="main"
Sep 11 16:50:58.986890 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.986815    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef648d29-870f-4bd8-bad2-b7d1445ccf63" containerName="main"
Sep 11 16:50:58.989891 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.989873    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:58.992321 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.992295    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 11 16:50:58.992443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.992295    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-f2kkw\""
Sep 11 16:50:58.992443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.992300    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 11 16:50:58.992443 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:58.992300    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 11 16:50:59.000277 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.000252    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"]
Sep 11 16:50:59.081265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.081235    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42733f32-bd2c-4ad9-b104-965673867208-tls-certs\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.081265 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.081266    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrwl\" (UniqueName: \"kubernetes.io/projected/42733f32-bd2c-4ad9-b104-965673867208-kube-api-access-mvrwl\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.081440 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.081287    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.081440 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.081311    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-home\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.081440 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.081333    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-dshm\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.081440 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.081355    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-model-cache\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.181819 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.181789    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42733f32-bd2c-4ad9-b104-965673867208-tls-certs\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.181955 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.181823    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrwl\" (UniqueName: \"kubernetes.io/projected/42733f32-bd2c-4ad9-b104-965673867208-kube-api-access-mvrwl\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.181955 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.181844    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.182029 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.181951    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-home\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.182029 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.181982    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-dshm\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.182029 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.182003    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-model-cache\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.182205 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.182187    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.182278 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.182262    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-home\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.182416 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.182389    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-model-cache\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.184248 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.184229    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-dshm\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.184587 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.184524    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42733f32-bd2c-4ad9-b104-965673867208-tls-certs\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.189742 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.189712    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrwl\" (UniqueName: \"kubernetes.io/projected/42733f32-bd2c-4ad9-b104-965673867208-kube-api-access-mvrwl\") pod \"test-div-v1-kserve-fb566d667-sftgt\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.300498 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.300436    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:50:59.574048 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:50:59.574016    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"]
Sep 11 16:50:59.577018 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:50:59.576976    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42733f32_bd2c_4ad9_b104_965673867208.slice/crio-fbdce2b02861e0daa8ea0a860c5710e141db2d7e07f8af2df6f5339c8595df78 WatchSource:0}: Error finding container fbdce2b02861e0daa8ea0a860c5710e141db2d7e07f8af2df6f5339c8595df78: Status 404 returned error can't find the container with id fbdce2b02861e0daa8ea0a860c5710e141db2d7e07f8af2df6f5339c8595df78
Sep 11 16:51:00.300110 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.300076    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" podUID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" containerName="main" containerID="cri-o://b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d" gracePeriod=45
Sep 11 16:51:00.343569 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.343527    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" podUID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 11 16:51:00.533878 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.533843    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:51:00.586557 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.586520    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt" event={"ID":"42733f32-bd2c-4ad9-b104-965673867208","Type":"ContainerStarted","Data":"8fcda083fdd1917239086c512cce5e048ece63b22e928e58c21bdcf9c1ff3fad"}
Sep 11 16:51:00.586723 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.586596    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt" event={"ID":"42733f32-bd2c-4ad9-b104-965673867208","Type":"ContainerStarted","Data":"fbdce2b02861e0daa8ea0a860c5710e141db2d7e07f8af2df6f5339c8595df78"}
Sep 11 16:51:00.587779 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.587757    2550 generic.go:356] "Generic (PLEG): container finished" podID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" containerID="b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d" exitCode=0
Sep 11 16:51:00.587883 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.587788    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" event={"ID":"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8","Type":"ContainerDied","Data":"b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d"}
Sep 11 16:51:00.587883 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.587799    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"
Sep 11 16:51:00.587883 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.587832    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b" event={"ID":"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8","Type":"ContainerDied","Data":"584bb06807dc7583da6a4e6072b5cf64f17c7138201f7372bc99dddd1a5722bc"}
Sep 11 16:51:00.587883 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.587854    2550 scope.go:117] "RemoveContainer" containerID="b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d"
Sep 11 16:51:00.594255 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594232    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx7qv\" (UniqueName: \"kubernetes.io/projected/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-kube-api-access-jx7qv\") pod \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") "
Sep 11 16:51:00.594354 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594259    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-home\") pod \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") "
Sep 11 16:51:00.594354 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594287    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-dshm\") pod \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") "
Sep 11 16:51:00.594480 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594458    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tmp-dir\") pod \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") "
Sep 11 16:51:00.594555 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594514    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-home" (OuterVolumeSpecName: "home") pod "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" (UID: "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:00.594555 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594530    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tls-certs\") pod \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") "
Sep 11 16:51:00.594680 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594579    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-model-cache\") pod \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\" (UID: \"862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8\") "
Sep 11 16:51:00.594732 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594701    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" (UID: "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:00.594903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594853    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:51:00.594903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594906    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:51:00.595081 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.594914    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" (UID: "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:00.596503 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.596484    2550 scope.go:117] "RemoveContainer" containerID="b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d"
Sep 11 16:51:00.596636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.596609    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-dshm" (OuterVolumeSpecName: "dshm") pod "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" (UID: "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:00.596715 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.596684    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" (UID: "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:51:00.596715 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.596693    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-kube-api-access-jx7qv" (OuterVolumeSpecName: "kube-api-access-jx7qv") pod "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" (UID: "862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8"). InnerVolumeSpecName "kube-api-access-jx7qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:51:00.596846 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:51:00.596826    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d\": container with ID starting with b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d not found: ID does not exist" containerID="b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d"
Sep 11 16:51:00.596885 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.596853    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d"} err="failed to get container status \"b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d\": rpc error: code = NotFound desc = could not find container \"b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d\": container with ID starting with b1756d79c0a954f5658a86057138eb98295bd426f7ce4814d696d2627705a41d not found: ID does not exist"
Sep 11 16:51:00.603602 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.603563    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt" podStartSLOduration=2.603536314 podStartE2EDuration="2.603536314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:00.601936742 +0000 UTC m=+2035.557984371" watchObservedRunningTime="2026-09-11 16:51:00.603536314 +0000 UTC m=+2035.559583941"
Sep 11 16:51:00.696141 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.696115    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:51:00.696141 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.696136    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:51:00.696141 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.696145    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7qv\" (UniqueName: \"kubernetes.io/projected/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-kube-api-access-jx7qv\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:51:00.696308 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.696153    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:51:00.911083 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.911050    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"]
Sep 11 16:51:00.917692 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:00.917669    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-xg62b"]
Sep 11 16:51:01.301435 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:01.301367    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:51:01.479433 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:01.479401    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" path="/var/lib/kubelet/pods/862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8/volumes"
Sep 11 16:51:09.300992 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.300956    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:51:09.313439 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.313415    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:51:09.391280 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.391248    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"]
Sep 11 16:51:09.391612 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.391600    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" containerName="main"
Sep 11 16:51:09.391664 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.391614    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" containerName="main"
Sep 11 16:51:09.391701 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.391677    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="862f9a49-2be2-4ef7-bb9e-7dd7c235bfc8" containerName="main"
Sep 11 16:51:09.396024 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.396004    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.398027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.398003    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 11 16:51:09.412116 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.412088    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"]
Sep 11 16:51:09.462129 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.462100    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89th\" (UniqueName: \"kubernetes.io/projected/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-kube-api-access-t89th\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.462244 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.462185    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.462416 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.462386    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-home\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.462528 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.462447    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.462528 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.462497    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.462638 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.462571    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563138 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563070    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563138 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563123    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-home\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563353 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563158    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563353 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563353 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563236    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563353 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563273    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89th\" (UniqueName: \"kubernetes.io/projected/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-kube-api-access-t89th\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563616 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563590    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-home\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563679 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563645    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.563679 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.563665    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.565501 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.565470    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.565790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.565770    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.572020 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.571995    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89th\" (UniqueName: \"kubernetes.io/projected/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-kube-api-access-t89th\") pod \"test-div-v2-kserve-5f6fdb6b77-qlsnk\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.641866 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.641846    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:51:09.705709 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.705678    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:09.977004 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:09.976976    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"]
Sep 11 16:51:09.978978 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:51:09.978947    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ec39d7_916d_4348_a6c1_0ba3a8fbca16.slice/crio-49662c36a9d0434b7db0e7d59fcd69bf32afe6656e0054562e8270b54352465d WatchSource:0}: Error finding container 49662c36a9d0434b7db0e7d59fcd69bf32afe6656e0054562e8270b54352465d: Status 404 returned error can't find the container with id 49662c36a9d0434b7db0e7d59fcd69bf32afe6656e0054562e8270b54352465d
Sep 11 16:51:10.638105 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:10.638064    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk" event={"ID":"33ec39d7-916d-4348-a6c1-0ba3a8fbca16","Type":"ContainerStarted","Data":"bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c"}
Sep 11 16:51:10.638504 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:10.638119    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk" event={"ID":"33ec39d7-916d-4348-a6c1-0ba3a8fbca16","Type":"ContainerStarted","Data":"49662c36a9d0434b7db0e7d59fcd69bf32afe6656e0054562e8270b54352465d"}
Sep 11 16:51:10.657239 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:10.657187    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk" podStartSLOduration=1.6571688660000001 podStartE2EDuration="1.657168866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:10.656336692 +0000 UTC m=+2045.612384319" watchObservedRunningTime="2026-09-11 16:51:10.657168866 +0000 UTC m=+2045.613216496"
Sep 11 16:51:10.705920 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:10.705889    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:19.706678 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:19.706647    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:19.719065 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:19.719042    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:20.695587 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:20.695529    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:51:24.632072 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.632026    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"]
Sep 11 16:51:24.635378 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.635345    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.637827 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.637794    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 11 16:51:24.638417 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.638302    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 11 16:51:24.638417 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.638328    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-57tm8\""
Sep 11 16:51:24.638631 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.638585    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 11 16:51:24.649417 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.649397    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"]
Sep 11 16:51:24.778935 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.778903    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.779113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.778959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.779113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.779040    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.779113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.779068    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.779113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.779090    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kms6v\" (UniqueName: \"kubernetes.io/projected/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-kube-api-access-kms6v\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.779294 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.779126    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880080 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880049    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880090    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880123    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880140    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880155    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kms6v\" (UniqueName: \"kubernetes.io/projected/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-kube-api-access-kms6v\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880228 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880182    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880538 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880513    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880630 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880586    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.880630 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.880607    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.882406 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.882350    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.882721 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.882704    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.887001 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.886979    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kms6v\" (UniqueName: \"kubernetes.io/projected/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-kube-api-access-kms6v\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-644489j66\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:24.947209 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:24.947183    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:25.224519 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:25.224488    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"]
Sep 11 16:51:25.710132 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:25.710099    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" event={"ID":"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a","Type":"ContainerStarted","Data":"3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"}
Sep 11 16:51:25.710635 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:25.710156    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" event={"ID":"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a","Type":"ContainerStarted","Data":"825d81e6f6d3e5ebb8f860433e6adc9d08b4ca2bf5767e9033e07f0f1a835daa"}
Sep 11 16:51:25.729488 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:25.729438    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" podStartSLOduration=1.729424013 podStartE2EDuration="1.729424013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:25.727861157 +0000 UTC m=+2060.683908786" watchObservedRunningTime="2026-09-11 16:51:25.729424013 +0000 UTC m=+2060.685471674"
Sep 11 16:51:25.947480 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:25.947448    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:34.947421 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:34.947348    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:34.960090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:34.960068    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:35.768198 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:35.768169    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:51:52.795653 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:52.795621    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"]
Sep 11 16:51:53.761234 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:51:53.761203    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"]
Sep 11 16:52:05.585761 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:05.585733    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:52:05.593948 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:05.593927    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:52:07.814049 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:07.813994    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk" podUID="33ec39d7-916d-4348-a6c1-0ba3a8fbca16" containerName="main" containerID="cri-o://bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c" gracePeriod=45
Sep 11 16:52:08.064217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.064162    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:52:08.113652 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113623    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-model-cache\") pod \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") "
Sep 11 16:52:08.113652 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113654    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tls-certs\") pod \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") "
Sep 11 16:52:08.113858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113689    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tmp-dir\") pod \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") "
Sep 11 16:52:08.113858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113749    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-dshm\") pod \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") "
Sep 11 16:52:08.113858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113777    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-home\") pod \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") "
Sep 11 16:52:08.113858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113796    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t89th\" (UniqueName: \"kubernetes.io/projected/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-kube-api-access-t89th\") pod \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\" (UID: \"33ec39d7-916d-4348-a6c1-0ba3a8fbca16\") "
Sep 11 16:52:08.114073 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.113885    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-model-cache" (OuterVolumeSpecName: "model-cache") pod "33ec39d7-916d-4348-a6c1-0ba3a8fbca16" (UID: "33ec39d7-916d-4348-a6c1-0ba3a8fbca16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:08.114073 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.114013    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33ec39d7-916d-4348-a6c1-0ba3a8fbca16" (UID: "33ec39d7-916d-4348-a6c1-0ba3a8fbca16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:08.114073 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.114022    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-home" (OuterVolumeSpecName: "home") pod "33ec39d7-916d-4348-a6c1-0ba3a8fbca16" (UID: "33ec39d7-916d-4348-a6c1-0ba3a8fbca16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:08.114208 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.114104    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:08.114208 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.114123    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:08.115886 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.115860    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-dshm" (OuterVolumeSpecName: "dshm") pod "33ec39d7-916d-4348-a6c1-0ba3a8fbca16" (UID: "33ec39d7-916d-4348-a6c1-0ba3a8fbca16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:08.115969 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.115928    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-kube-api-access-t89th" (OuterVolumeSpecName: "kube-api-access-t89th") pod "33ec39d7-916d-4348-a6c1-0ba3a8fbca16" (UID: "33ec39d7-916d-4348-a6c1-0ba3a8fbca16"). InnerVolumeSpecName "kube-api-access-t89th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:52:08.115969 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.115941    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33ec39d7-916d-4348-a6c1-0ba3a8fbca16" (UID: "33ec39d7-916d-4348-a6c1-0ba3a8fbca16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:52:08.181316 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.181283    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"]
Sep 11 16:52:08.181656 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.181642    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ec39d7-916d-4348-a6c1-0ba3a8fbca16" containerName="main"
Sep 11 16:52:08.181716 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.181657    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ec39d7-916d-4348-a6c1-0ba3a8fbca16" containerName="main"
Sep 11 16:52:08.181761 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.181721    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ec39d7-916d-4348-a6c1-0ba3a8fbca16" containerName="main"
Sep 11 16:52:08.184884 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.184861    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.187030 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.186999    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 11 16:52:08.187156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.187030    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-xpgxc\""
Sep 11 16:52:08.187156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.187039    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 11 16:52:08.187156 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.187107    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 11 16:52:08.194832 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.194809    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"]
Sep 11 16:52:08.215426 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.215403    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:08.215426 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.215421    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:08.215586 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.215432    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t89th\" (UniqueName: \"kubernetes.io/projected/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-kube-api-access-t89th\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:08.215586 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.215441    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33ec39d7-916d-4348-a6c1-0ba3a8fbca16-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:08.316790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.316730    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-model-cache\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.316790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.316762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tls-certs\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.316921 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.316793    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-home\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.316921 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.316808    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-dshm\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.316921 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.316827    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxf75\" (UniqueName: \"kubernetes.io/projected/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-kube-api-access-qxf75\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.316921 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.316861    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417353    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417405    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-model-cache\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417439    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tls-certs\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417483    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-home\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417521 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417502    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-dshm\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417764 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417531    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxf75\" (UniqueName: \"kubernetes.io/projected/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-kube-api-access-qxf75\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417810    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417895 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417838    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-model-cache\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.417895 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.417883    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-home\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.419754 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.419734    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-dshm\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.419983 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.419966    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tls-certs\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.424717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.424696    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxf75\" (UniqueName: \"kubernetes.io/projected/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-kube-api-access-qxf75\") pod \"stop-v1-kserve-5657b88c95-hrmmw\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.496382 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.496358    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:08.766715 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.766690    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"]
Sep 11 16:52:08.769427 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:52:08.769398    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83f4583_be6e_4534_bedc_a72cd1fcc9ce.slice/crio-7083c25d4d17751c67ec08341845741bd516c788fe73fe21588887d67fd236aa WatchSource:0}: Error finding container 7083c25d4d17751c67ec08341845741bd516c788fe73fe21588887d67fd236aa: Status 404 returned error can't find the container with id 7083c25d4d17751c67ec08341845741bd516c788fe73fe21588887d67fd236aa
Sep 11 16:52:08.782363 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.782311    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt" podUID="42733f32-bd2c-4ad9-b104-965673867208" containerName="main" containerID="cri-o://8fcda083fdd1917239086c512cce5e048ece63b22e928e58c21bdcf9c1ff3fad" gracePeriod=45
Sep 11 16:52:08.926393 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.926355    2550 generic.go:356] "Generic (PLEG): container finished" podID="42733f32-bd2c-4ad9-b104-965673867208" containerID="8fcda083fdd1917239086c512cce5e048ece63b22e928e58c21bdcf9c1ff3fad" exitCode=0
Sep 11 16:52:08.926946 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.926473    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt" event={"ID":"42733f32-bd2c-4ad9-b104-965673867208","Type":"ContainerDied","Data":"8fcda083fdd1917239086c512cce5e048ece63b22e928e58c21bdcf9c1ff3fad"}
Sep 11 16:52:08.928194 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.928168    2550 generic.go:356] "Generic (PLEG): container finished" podID="33ec39d7-916d-4348-a6c1-0ba3a8fbca16" containerID="bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c" exitCode=0
Sep 11 16:52:08.928299 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.928199    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk" event={"ID":"33ec39d7-916d-4348-a6c1-0ba3a8fbca16","Type":"ContainerDied","Data":"bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c"}
Sep 11 16:52:08.928299 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.928231    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"
Sep 11 16:52:08.928299 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.928252    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk" event={"ID":"33ec39d7-916d-4348-a6c1-0ba3a8fbca16","Type":"ContainerDied","Data":"49662c36a9d0434b7db0e7d59fcd69bf32afe6656e0054562e8270b54352465d"}
Sep 11 16:52:08.928299 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.928279    2550 scope.go:117] "RemoveContainer" containerID="bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c"
Sep 11 16:52:08.929996 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.929972    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw" event={"ID":"e83f4583-be6e-4534-bedc-a72cd1fcc9ce","Type":"ContainerStarted","Data":"3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72"}
Sep 11 16:52:08.930113 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.930015    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw" event={"ID":"e83f4583-be6e-4534-bedc-a72cd1fcc9ce","Type":"ContainerStarted","Data":"7083c25d4d17751c67ec08341845741bd516c788fe73fe21588887d67fd236aa"}
Sep 11 16:52:08.937486 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.937462    2550 scope.go:117] "RemoveContainer" containerID="bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c"
Sep 11 16:52:08.937784 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:52:08.937763    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c\": container with ID starting with bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c not found: ID does not exist" containerID="bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c"
Sep 11 16:52:08.937885 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.937793    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c"} err="failed to get container status \"bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c\": rpc error: code = NotFound desc = could not find container \"bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c\": container with ID starting with bf9112833ee4e0fc5efb00840c2a6d3b6a501e38b4377e8b87ab84a358e7df5c not found: ID does not exist"
Sep 11 16:52:08.949428 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.949379    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw" podStartSLOduration=0.94936271 podStartE2EDuration="949.36271ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:08.948617556 +0000 UTC m=+2103.904665188" watchObservedRunningTime="2026-09-11 16:52:08.94936271 +0000 UTC m=+2103.905410343"
Sep 11 16:52:08.964103 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.964076    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"]
Sep 11 16:52:08.967242 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:08.967219    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-qlsnk"]
Sep 11 16:52:09.043874 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.043846    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:52:09.123282 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123252    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-tmp-dir\") pod \"42733f32-bd2c-4ad9-b104-965673867208\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") "
Sep 11 16:52:09.123282 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123286    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-dshm\") pod \"42733f32-bd2c-4ad9-b104-965673867208\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") "
Sep 11 16:52:09.123480 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123316    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-home\") pod \"42733f32-bd2c-4ad9-b104-965673867208\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") "
Sep 11 16:52:09.123480 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123340    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42733f32-bd2c-4ad9-b104-965673867208-tls-certs\") pod \"42733f32-bd2c-4ad9-b104-965673867208\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") "
Sep 11 16:52:09.123480 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123365    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvrwl\" (UniqueName: \"kubernetes.io/projected/42733f32-bd2c-4ad9-b104-965673867208-kube-api-access-mvrwl\") pod \"42733f32-bd2c-4ad9-b104-965673867208\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") "
Sep 11 16:52:09.123480 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123430    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-model-cache\") pod \"42733f32-bd2c-4ad9-b104-965673867208\" (UID: \"42733f32-bd2c-4ad9-b104-965673867208\") "
Sep 11 16:52:09.123785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123592    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42733f32-bd2c-4ad9-b104-965673867208" (UID: "42733f32-bd2c-4ad9-b104-965673867208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:09.123785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123619    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-home" (OuterVolumeSpecName: "home") pod "42733f32-bd2c-4ad9-b104-965673867208" (UID: "42733f32-bd2c-4ad9-b104-965673867208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:09.123785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123719    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:09.123785 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123733    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:09.123942 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.123798    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-model-cache" (OuterVolumeSpecName: "model-cache") pod "42733f32-bd2c-4ad9-b104-965673867208" (UID: "42733f32-bd2c-4ad9-b104-965673867208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:09.125453 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.125432    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-dshm" (OuterVolumeSpecName: "dshm") pod "42733f32-bd2c-4ad9-b104-965673867208" (UID: "42733f32-bd2c-4ad9-b104-965673867208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:52:09.125569 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.125445    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42733f32-bd2c-4ad9-b104-965673867208-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42733f32-bd2c-4ad9-b104-965673867208" (UID: "42733f32-bd2c-4ad9-b104-965673867208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:52:09.125569 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.125558    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42733f32-bd2c-4ad9-b104-965673867208-kube-api-access-mvrwl" (OuterVolumeSpecName: "kube-api-access-mvrwl") pod "42733f32-bd2c-4ad9-b104-965673867208" (UID: "42733f32-bd2c-4ad9-b104-965673867208"). InnerVolumeSpecName "kube-api-access-mvrwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:52:09.224827 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.224795    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:09.224827 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.224821    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42733f32-bd2c-4ad9-b104-965673867208-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:09.224827 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.224832    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvrwl\" (UniqueName: \"kubernetes.io/projected/42733f32-bd2c-4ad9-b104-965673867208-kube-api-access-mvrwl\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:09.225039 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.224840    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42733f32-bd2c-4ad9-b104-965673867208-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:52:09.477611 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.477581    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ec39d7-916d-4348-a6c1-0ba3a8fbca16" path="/var/lib/kubelet/pods/33ec39d7-916d-4348-a6c1-0ba3a8fbca16/volumes"
Sep 11 16:52:09.497031 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.497011    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:09.938838 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.938805    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt" event={"ID":"42733f32-bd2c-4ad9-b104-965673867208","Type":"ContainerDied","Data":"fbdce2b02861e0daa8ea0a860c5710e141db2d7e07f8af2df6f5339c8595df78"}
Sep 11 16:52:09.938838 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.938838    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"
Sep 11 16:52:09.939339 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.938865    2550 scope.go:117] "RemoveContainer" containerID="8fcda083fdd1917239086c512cce5e048ece63b22e928e58c21bdcf9c1ff3fad"
Sep 11 16:52:09.953525 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.953503    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"]
Sep 11 16:52:09.958740 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:09.958720    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-sftgt"]
Sep 11 16:52:11.477121 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:11.477086    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42733f32-bd2c-4ad9-b104-965673867208" path="/var/lib/kubelet/pods/42733f32-bd2c-4ad9-b104-965673867208/volumes"
Sep 11 16:52:16.669258 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.669226    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"]
Sep 11 16:52:16.669650 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.669595    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42733f32-bd2c-4ad9-b104-965673867208" containerName="main"
Sep 11 16:52:16.669650 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.669620    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="42733f32-bd2c-4ad9-b104-965673867208" containerName="main"
Sep 11 16:52:16.669765 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.669733    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="42733f32-bd2c-4ad9-b104-965673867208" containerName="main"
Sep 11 16:52:16.672689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.672669    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.674743 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.674722    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 11 16:52:16.682602 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.682577    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"]
Sep 11 16:52:16.782127 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.782091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-model-cache\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.782127 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.782130    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-home\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.782350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.782161    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tls-certs\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.782350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.782210    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.782350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.782274    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvll\" (UniqueName: \"kubernetes.io/projected/ebf0d6e4-932f-4ff6-a456-44e83ae029db-kube-api-access-wsvll\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.782350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.782302    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-dshm\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.882666 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.882636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.882797 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.882689    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvll\" (UniqueName: \"kubernetes.io/projected/ebf0d6e4-932f-4ff6-a456-44e83ae029db-kube-api-access-wsvll\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.882797 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.882719    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-dshm\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.882797 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.882756    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-model-cache\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.882797 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.882781    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-home\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.883028 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.882826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tls-certs\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.883028 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.883005    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.883134 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.883114    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-model-cache\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.883188 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.883136    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-home\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.884911 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.884890    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-dshm\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.885278 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.885259    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tls-certs\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.889584 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.889567    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvll\" (UniqueName: \"kubernetes.io/projected/ebf0d6e4-932f-4ff6-a456-44e83ae029db-kube-api-access-wsvll\") pod \"stop-v2-kserve-7f68747c45-6wq2s\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:16.984150 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:16.984085    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:17.255790 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:17.255711    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"]
Sep 11 16:52:17.259395 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:52:17.259372    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf0d6e4_932f_4ff6_a456_44e83ae029db.slice/crio-243be5e7a105cae13480ffc19981ef1f560b13e25eebedfad8e3419d6c43bfbb WatchSource:0}: Error finding container 243be5e7a105cae13480ffc19981ef1f560b13e25eebedfad8e3419d6c43bfbb: Status 404 returned error can't find the container with id 243be5e7a105cae13480ffc19981ef1f560b13e25eebedfad8e3419d6c43bfbb
Sep 11 16:52:17.978244 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:17.978202    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s" event={"ID":"ebf0d6e4-932f-4ff6-a456-44e83ae029db","Type":"ContainerStarted","Data":"d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b"}
Sep 11 16:52:17.978617 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:17.978254    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s" event={"ID":"ebf0d6e4-932f-4ff6-a456-44e83ae029db","Type":"ContainerStarted","Data":"243be5e7a105cae13480ffc19981ef1f560b13e25eebedfad8e3419d6c43bfbb"}
Sep 11 16:52:17.985106 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:17.985086    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:17.994709 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:17.994663    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s" podStartSLOduration=1.994649973 podStartE2EDuration="1.994649973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:52:17.992898369 +0000 UTC m=+2112.948945990" watchObservedRunningTime="2026-09-11 16:52:17.994649973 +0000 UTC m=+2112.950697600"
Sep 11 16:52:18.496601 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:18.496572    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:18.509186 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:18.509157    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:18.993748 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:18.993720    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:52:26.984710 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:26.984675    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:26.997227 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:26.997200    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:27.038598 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:27.038573    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:52:59.403809 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:59.403724    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"]
Sep 11 16:52:59.412307 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:52:59.412273    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:52:59.412307 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:52:59.412307 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:52:59.412307 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7" cmd=["/bin/sleep","15"]
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:59.412322    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" message="" err=<
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:52:59.412334    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerName="main" containerID="cri-o://3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"
Sep 11 16:52:59.412533 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:59.412387    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerName="main" containerID="cri-o://3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7" gracePeriod=60
Sep 11 16:52:59.757733 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:52:59.757649    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 11 16:53:00.154533 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.154510    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:53:00.186381 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.186347    2550 generic.go:356] "Generic (PLEG): container finished" podID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerID="3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7" exitCode=0
Sep 11 16:53:00.186505 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.186385    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"
Sep 11 16:53:00.186505 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.186399    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" event={"ID":"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a","Type":"ContainerDied","Data":"3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"}
Sep 11 16:53:00.186505 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.186451    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66" event={"ID":"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a","Type":"ContainerDied","Data":"825d81e6f6d3e5ebb8f860433e6adc9d08b4ca2bf5767e9033e07f0f1a835daa"}
Sep 11 16:53:00.186642 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.186483    2550 scope.go:117] "RemoveContainer" containerID="3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"
Sep 11 16:53:00.195249 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.195218    2550 scope.go:117] "RemoveContainer" containerID="3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"
Sep 11 16:53:00.195523 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:53:00.195500    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7\": container with ID starting with 3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7 not found: ID does not exist" containerID="3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"
Sep 11 16:53:00.195646 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.195530    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7"} err="failed to get container status \"3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7\": rpc error: code = NotFound desc = could not find container \"3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7\": container with ID starting with 3d0291af18035e4173ef376b4db477cb120650aad081822f49437ffe7ee76eb7 not found: ID does not exist"
Sep 11 16:53:00.209735 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209717    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-home\") pod \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") "
Sep 11 16:53:00.209828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209768    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-dshm\") pod \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") "
Sep 11 16:53:00.209828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209792    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kms6v\" (UniqueName: \"kubernetes.io/projected/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-kube-api-access-kms6v\") pod \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") "
Sep 11 16:53:00.209828 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209816    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-model-cache\") pod \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") "
Sep 11 16:53:00.209940 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209866    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tls-certs\") pod \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") "
Sep 11 16:53:00.209940 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209907    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tmp-dir\") pod \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\" (UID: \"6fc46647-4ee5-4b4a-92ce-dbaac789cd9a\") "
Sep 11 16:53:00.210024 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.209971    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-home" (OuterVolumeSpecName: "home") pod "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" (UID: "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.211529 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.210489    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" (UID: "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.211529 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.210566    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" (UID: "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.211529 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.210893    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.214138 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.214113    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-kube-api-access-kms6v" (OuterVolumeSpecName: "kube-api-access-kms6v") pod "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" (UID: "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a"). InnerVolumeSpecName "kube-api-access-kms6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:53:00.217481 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.217458    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-dshm" (OuterVolumeSpecName: "dshm") pod "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" (UID: "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:00.218314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.218294    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" (UID: "6fc46647-4ee5-4b4a-92ce-dbaac789cd9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:53:00.312119 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.312056    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.312119 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.312080    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kms6v\" (UniqueName: \"kubernetes.io/projected/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-kube-api-access-kms6v\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.312119 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.312091    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.312119 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.312099    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.312119 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.312107    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:00.505969 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.505940    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"]
Sep 11 16:53:00.509683 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:00.509661    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-644489j66"]
Sep 11 16:53:01.476718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:01.476686    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" path="/var/lib/kubelet/pods/6fc46647-4ee5-4b4a-92ce-dbaac789cd9a/volumes"
Sep 11 16:53:18.345671 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:18.345637    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"]
Sep 11 16:53:33.366682 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.366563    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s" podUID="ebf0d6e4-932f-4ff6-a456-44e83ae029db" containerName="main" containerID="cri-o://d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b" gracePeriod=45
Sep 11 16:53:33.619765 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.619714    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:53:33.774747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.774716    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tmp-dir\") pod \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") "
Sep 11 16:53:33.774903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.774752    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsvll\" (UniqueName: \"kubernetes.io/projected/ebf0d6e4-932f-4ff6-a456-44e83ae029db-kube-api-access-wsvll\") pod \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") "
Sep 11 16:53:33.774903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.774781    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-model-cache\") pod \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") "
Sep 11 16:53:33.774903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.774803    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-dshm\") pod \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") "
Sep 11 16:53:33.774903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.774819    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-home\") pod \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") "
Sep 11 16:53:33.774903 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.774853    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tls-certs\") pod \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\" (UID: \"ebf0d6e4-932f-4ff6-a456-44e83ae029db\") "
Sep 11 16:53:33.775180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.775043    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebf0d6e4-932f-4ff6-a456-44e83ae029db" (UID: "ebf0d6e4-932f-4ff6-a456-44e83ae029db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:33.775180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.775049    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebf0d6e4-932f-4ff6-a456-44e83ae029db" (UID: "ebf0d6e4-932f-4ff6-a456-44e83ae029db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:33.775180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.775112    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-home" (OuterVolumeSpecName: "home") pod "ebf0d6e4-932f-4ff6-a456-44e83ae029db" (UID: "ebf0d6e4-932f-4ff6-a456-44e83ae029db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:33.777126 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.777099    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebf0d6e4-932f-4ff6-a456-44e83ae029db" (UID: "ebf0d6e4-932f-4ff6-a456-44e83ae029db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:53:33.777234 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.777115    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-dshm" (OuterVolumeSpecName: "dshm") pod "ebf0d6e4-932f-4ff6-a456-44e83ae029db" (UID: "ebf0d6e4-932f-4ff6-a456-44e83ae029db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:33.777234 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.777141    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf0d6e4-932f-4ff6-a456-44e83ae029db-kube-api-access-wsvll" (OuterVolumeSpecName: "kube-api-access-wsvll") pod "ebf0d6e4-932f-4ff6-a456-44e83ae029db" (UID: "ebf0d6e4-932f-4ff6-a456-44e83ae029db"). InnerVolumeSpecName "kube-api-access-wsvll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:53:33.875882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.875826    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:33.875882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.875845    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:33.875882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.875853    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:33.875882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.875862    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:33.875882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.875870    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsvll\" (UniqueName: \"kubernetes.io/projected/ebf0d6e4-932f-4ff6-a456-44e83ae029db-kube-api-access-wsvll\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:33.875882 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:33.875878    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf0d6e4-932f-4ff6-a456-44e83ae029db-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:34.361381 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.361351    2550 generic.go:356] "Generic (PLEG): container finished" podID="ebf0d6e4-932f-4ff6-a456-44e83ae029db" containerID="d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b" exitCode=0
Sep 11 16:53:34.361537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.361399    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"
Sep 11 16:53:34.361537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.361438    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s" event={"ID":"ebf0d6e4-932f-4ff6-a456-44e83ae029db","Type":"ContainerDied","Data":"d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b"}
Sep 11 16:53:34.361537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.361495    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s" event={"ID":"ebf0d6e4-932f-4ff6-a456-44e83ae029db","Type":"ContainerDied","Data":"243be5e7a105cae13480ffc19981ef1f560b13e25eebedfad8e3419d6c43bfbb"}
Sep 11 16:53:34.361537 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.361525    2550 scope.go:117] "RemoveContainer" containerID="d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b"
Sep 11 16:53:34.370570 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.370367    2550 scope.go:117] "RemoveContainer" containerID="d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b"
Sep 11 16:53:34.370809 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:53:34.370601    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b\": container with ID starting with d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b not found: ID does not exist" containerID="d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b"
Sep 11 16:53:34.370809 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.370622    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b"} err="failed to get container status \"d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b\": rpc error: code = NotFound desc = could not find container \"d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b\": container with ID starting with d2715575e6c934814e832c93ff7482006ec9f9ce92ce8ee642af5530a2eb8b3b not found: ID does not exist"
Sep 11 16:53:34.384383 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.384361    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"]
Sep 11 16:53:34.388049 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:34.388028    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-6wq2s"]
Sep 11 16:53:35.476953 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:35.476922    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf0d6e4-932f-4ff6-a456-44e83ae029db" path="/var/lib/kubelet/pods/ebf0d6e4-932f-4ff6-a456-44e83ae029db/volumes"
Sep 11 16:53:39.995722 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:39.995688    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"]
Sep 11 16:53:47.776847 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.776815    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"]
Sep 11 16:53:47.777314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.777134    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf0d6e4-932f-4ff6-a456-44e83ae029db" containerName="main"
Sep 11 16:53:47.777314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.777144    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf0d6e4-932f-4ff6-a456-44e83ae029db" containerName="main"
Sep 11 16:53:47.777314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.777158    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerName="main"
Sep 11 16:53:47.777314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.777163    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerName="main"
Sep 11 16:53:47.777314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.777214    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf0d6e4-932f-4ff6-a456-44e83ae029db" containerName="main"
Sep 11 16:53:47.777314 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.777221    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc46647-4ee5-4b4a-92ce-dbaac789cd9a" containerName="main"
Sep 11 16:53:47.779975 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.779955    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.782613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.782587    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 11 16:53:47.782613 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.782603    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 11 16:53:47.782801 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.782621    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 11 16:53:47.782801 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.782604    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-r6v2b\""
Sep 11 16:53:47.790078 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.790051    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"]
Sep 11 16:53:47.884196 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.884163    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.884343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.884208    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cfa438-9f02-4369-83ab-83f89a074ddf-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.884343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.884232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-home\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.884343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.884246    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-model-cache\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.884343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.884264    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgcds\" (UniqueName: \"kubernetes.io/projected/e9cfa438-9f02-4369-83ab-83f89a074ddf-kube-api-access-hgcds\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.884343 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.884338    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-dshm\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.984947 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.984914    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcds\" (UniqueName: \"kubernetes.io/projected/e9cfa438-9f02-4369-83ab-83f89a074ddf-kube-api-access-hgcds\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.984957    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-dshm\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985013    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985051    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cfa438-9f02-4369-83ab-83f89a074ddf-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985231 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985089    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-home\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985231 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985112    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-model-cache\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985460 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985422    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985562 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985440    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-model-cache\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.985612 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.985575    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-home\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.987320 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.987303    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-dshm\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.987805 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.987788    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cfa438-9f02-4369-83ab-83f89a074ddf-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:47.991625 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:47.991605    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcds\" (UniqueName: \"kubernetes.io/projected/e9cfa438-9f02-4369-83ab-83f89a074ddf-kube-api-access-hgcds\") pod \"decom-v1-kserve-6cb797fb75-xpwdh\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:48.090638 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:48.090614    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:48.368871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:48.368810    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"]
Sep 11 16:53:48.370958 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:53:48.370927    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cfa438_9f02_4369_83ab_83f89a074ddf.slice/crio-0c47fa9151dedfec5fd54d756a0f9d14fd5c216b060bde4d58fdcfd86b61371a WatchSource:0}: Error finding container 0c47fa9151dedfec5fd54d756a0f9d14fd5c216b060bde4d58fdcfd86b61371a: Status 404 returned error can't find the container with id 0c47fa9151dedfec5fd54d756a0f9d14fd5c216b060bde4d58fdcfd86b61371a
Sep 11 16:53:48.436181 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:48.436157    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh" event={"ID":"e9cfa438-9f02-4369-83ab-83f89a074ddf","Type":"ContainerStarted","Data":"4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71"}
Sep 11 16:53:48.436295 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:48.436210    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh" event={"ID":"e9cfa438-9f02-4369-83ab-83f89a074ddf","Type":"ContainerStarted","Data":"0c47fa9151dedfec5fd54d756a0f9d14fd5c216b060bde4d58fdcfd86b61371a"}
Sep 11 16:53:48.456942 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:48.456884    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh" podStartSLOduration=1.456864679 podStartE2EDuration="1.456864679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:53:48.452120363 +0000 UTC m=+2203.408167991" watchObservedRunningTime="2026-09-11 16:53:48.456864679 +0000 UTC m=+2203.412912307"
Sep 11 16:53:49.091468 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:49.091437    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:55.045347 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.045304    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw" podUID="e83f4583-be6e-4534-bedc-a72cd1fcc9ce" containerName="main" containerID="cri-o://3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72" gracePeriod=45
Sep 11 16:53:55.295291 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.295270    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:53:55.442960 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.442933    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tls-certs\") pod \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") "
Sep 11 16:53:55.443101 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.442970    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-model-cache\") pod \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") "
Sep 11 16:53:55.443101 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.442988    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-dshm\") pod \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") "
Sep 11 16:53:55.443177 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443131    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tmp-dir\") pod \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") "
Sep 11 16:53:55.443222 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443188    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxf75\" (UniqueName: \"kubernetes.io/projected/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-kube-api-access-qxf75\") pod \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") "
Sep 11 16:53:55.443222 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443206    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-home\") pod \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\" (UID: \"e83f4583-be6e-4534-bedc-a72cd1fcc9ce\") "
Sep 11 16:53:55.443307 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443241    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "e83f4583-be6e-4534-bedc-a72cd1fcc9ce" (UID: "e83f4583-be6e-4534-bedc-a72cd1fcc9ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:55.443369 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443349    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e83f4583-be6e-4534-bedc-a72cd1fcc9ce" (UID: "e83f4583-be6e-4534-bedc-a72cd1fcc9ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:55.443470 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443449    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-home" (OuterVolumeSpecName: "home") pod "e83f4583-be6e-4534-bedc-a72cd1fcc9ce" (UID: "e83f4583-be6e-4534-bedc-a72cd1fcc9ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:55.443535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443465    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:55.443535 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.443486    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:55.445142 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.445113    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e83f4583-be6e-4534-bedc-a72cd1fcc9ce" (UID: "e83f4583-be6e-4534-bedc-a72cd1fcc9ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:53:55.445142 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.445129    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-dshm" (OuterVolumeSpecName: "dshm") pod "e83f4583-be6e-4534-bedc-a72cd1fcc9ce" (UID: "e83f4583-be6e-4534-bedc-a72cd1fcc9ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:53:55.445282 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.445208    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-kube-api-access-qxf75" (OuterVolumeSpecName: "kube-api-access-qxf75") pod "e83f4583-be6e-4534-bedc-a72cd1fcc9ce" (UID: "e83f4583-be6e-4534-bedc-a72cd1fcc9ce"). InnerVolumeSpecName "kube-api-access-qxf75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:53:55.471351 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.471323    2550 generic.go:356] "Generic (PLEG): container finished" podID="e83f4583-be6e-4534-bedc-a72cd1fcc9ce" containerID="3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72" exitCode=0
Sep 11 16:53:55.473618 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.473595    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"
Sep 11 16:53:55.477307 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.477280    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw" event={"ID":"e83f4583-be6e-4534-bedc-a72cd1fcc9ce","Type":"ContainerDied","Data":"3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72"}
Sep 11 16:53:55.477404 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.477331    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw" event={"ID":"e83f4583-be6e-4534-bedc-a72cd1fcc9ce","Type":"ContainerDied","Data":"7083c25d4d17751c67ec08341845741bd516c788fe73fe21588887d67fd236aa"}
Sep 11 16:53:55.477404 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.477352    2550 scope.go:117] "RemoveContainer" containerID="3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72"
Sep 11 16:53:55.486471 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.486450    2550 scope.go:117] "RemoveContainer" containerID="3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72"
Sep 11 16:53:55.486774 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:53:55.486755    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72\": container with ID starting with 3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72 not found: ID does not exist" containerID="3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72"
Sep 11 16:53:55.486832 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.486781    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72"} err="failed to get container status \"3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72\": rpc error: code = NotFound desc = could not find container \"3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72\": container with ID starting with 3ccb4f1415687cb80ac8bd8edbab0caadbdf27b570212c86acac99641aef1b72 not found: ID does not exist"
Sep 11 16:53:55.496703 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.496684    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"]
Sep 11 16:53:55.501377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.501355    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-hrmmw"]
Sep 11 16:53:55.544468 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.544451    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:55.544468 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.544468    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxf75\" (UniqueName: \"kubernetes.io/projected/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-kube-api-access-qxf75\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:55.544610 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.544477    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:55.544610 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:55.544486    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e83f4583-be6e-4534-bedc-a72cd1fcc9ce-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:53:57.476921 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:57.476888    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83f4583-be6e-4534-bedc-a72cd1fcc9ce" path="/var/lib/kubelet/pods/e83f4583-be6e-4534-bedc-a72cd1fcc9ce/volumes"
Sep 11 16:53:58.091208 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:58.091173    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:58.103844 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:58.103821    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:53:58.498426 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:53:58.498341    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:54:04.873803 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.873661    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"]
Sep 11 16:54:04.874434 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.874145    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83f4583-be6e-4534-bedc-a72cd1fcc9ce" containerName="main"
Sep 11 16:54:04.874434 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.874162    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83f4583-be6e-4534-bedc-a72cd1fcc9ce" containerName="main"
Sep 11 16:54:04.874434 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.874269    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83f4583-be6e-4534-bedc-a72cd1fcc9ce" containerName="main"
Sep 11 16:54:04.878497 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.878473    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:04.880452 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.880431    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 11 16:54:04.885147 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.885123    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"]
Sep 11 16:54:04.909140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.909120    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-model-cache\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:04.909241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.909151    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a80c96b-e2d2-4259-9e93-d0328685ceba-tls-certs\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:04.909241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.909170    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npt6g\" (UniqueName: \"kubernetes.io/projected/2a80c96b-e2d2-4259-9e93-d0328685ceba-kube-api-access-npt6g\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:04.909241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.909191    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-tmp-dir\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:04.909350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.909259    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-home\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:04.909350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:04.909316    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-dshm\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010467 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010437    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-dshm\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010482    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-model-cache\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a80c96b-e2d2-4259-9e93-d0328685ceba-tls-certs\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010537    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npt6g\" (UniqueName: \"kubernetes.io/projected/2a80c96b-e2d2-4259-9e93-d0328685ceba-kube-api-access-npt6g\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010574    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-tmp-dir\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010623 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010593    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-home\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.010966 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010945    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-model-cache\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.011036 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010977    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-home\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.011036 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.010997    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-tmp-dir\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.012746 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.012726    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-dshm\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.012965 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.012947    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a80c96b-e2d2-4259-9e93-d0328685ceba-tls-certs\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.018747 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.018729    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npt6g\" (UniqueName: \"kubernetes.io/projected/2a80c96b-e2d2-4259-9e93-d0328685ceba-kube-api-access-npt6g\") pod \"decom-v2-kserve-77f867d787-lwdcc\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.192806 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.192740    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:05.464079 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.464053    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"]
Sep 11 16:54:05.465620 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:54:05.465595    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a80c96b_e2d2_4259_9e93_d0328685ceba.slice/crio-e434868cb9b5d6c6501f8e5830eef3283c2a7705c94d2e6ef026fd3c3dda81f9 WatchSource:0}: Error finding container e434868cb9b5d6c6501f8e5830eef3283c2a7705c94d2e6ef026fd3c3dda81f9: Status 404 returned error can't find the container with id e434868cb9b5d6c6501f8e5830eef3283c2a7705c94d2e6ef026fd3c3dda81f9
Sep 11 16:54:05.522305 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:05.522275    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" event={"ID":"2a80c96b-e2d2-4259-9e93-d0328685ceba","Type":"ContainerStarted","Data":"e434868cb9b5d6c6501f8e5830eef3283c2a7705c94d2e6ef026fd3c3dda81f9"}
Sep 11 16:54:06.528332 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:06.528296    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" event={"ID":"2a80c96b-e2d2-4259-9e93-d0328685ceba","Type":"ContainerStarted","Data":"f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9"}
Sep 11 16:54:06.546317 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:06.546263    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" podStartSLOduration=2.546245039 podStartE2EDuration="2.546245039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:54:06.54383416 +0000 UTC m=+2221.499881789" watchObservedRunningTime="2026-09-11 16:54:06.546245039 +0000 UTC m=+2221.502292668"
Sep 11 16:54:07.193492 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:07.193462    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:15.193865 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:15.193832    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:15.206437 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:15.206413    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:15.577601 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:15.577509    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:54:31.064814 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.064738    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"]
Sep 11 16:54:31.068579 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.068557    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.071216 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.071193    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-djghm\""
Sep 11 16:54:31.071394 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.071228    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 11 16:54:31.071394 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.071291    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 11 16:54:31.071394 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.071318    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-mzqfb\""
Sep 11 16:54:31.071394 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.071360    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 11 16:54:31.078760 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.078731    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"]
Sep 11 16:54:31.108006 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.107983    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kclh\" (UniqueName: \"kubernetes.io/projected/ff8e4faa-2da6-4fef-a461-87693c9ae87a-kube-api-access-9kclh\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.108107 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.108038    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8e4faa-2da6-4fef-a461-87693c9ae87a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.209277 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.209250    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kclh\" (UniqueName: \"kubernetes.io/projected/ff8e4faa-2da6-4fef-a461-87693c9ae87a-kube-api-access-9kclh\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.209386 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.209302    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8e4faa-2da6-4fef-a461-87693c9ae87a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.211798 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.211781    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8e4faa-2da6-4fef-a461-87693c9ae87a-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.216793 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.216772    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kclh\" (UniqueName: \"kubernetes.io/projected/ff8e4faa-2da6-4fef-a461-87693c9ae87a-kube-api-access-9kclh\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.380175 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.380144    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:31.652367 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:31.652345    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"]
Sep 11 16:54:31.654230 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:54:31.654199    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8e4faa_2da6_4fef_a461_87693c9ae87a.slice/crio-d2f5ed6c04b593c9797e37db3f29728adfc13721591e7ff902b228d535798ee0 WatchSource:0}: Error finding container d2f5ed6c04b593c9797e37db3f29728adfc13721591e7ff902b228d535798ee0: Status 404 returned error can't find the container with id d2f5ed6c04b593c9797e37db3f29728adfc13721591e7ff902b228d535798ee0
Sep 11 16:54:32.648821 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:32.648771    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" event={"ID":"ff8e4faa-2da6-4fef-a461-87693c9ae87a","Type":"ContainerStarted","Data":"65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27"}
Sep 11 16:54:32.649217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:32.648848    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:54:32.649217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:32.648873    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" event={"ID":"ff8e4faa-2da6-4fef-a461-87693c9ae87a","Type":"ContainerStarted","Data":"d2f5ed6c04b593c9797e37db3f29728adfc13721591e7ff902b228d535798ee0"}
Sep 11 16:54:32.665474 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:54:32.665424    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podStartSLOduration=1.6654088950000001 podStartE2EDuration="1.665408895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:54:32.663663365 +0000 UTC m=+2247.619711003" watchObservedRunningTime="2026-09-11 16:54:32.665408895 +0000 UTC m=+2247.621456522"
Sep 11 16:55:03.656401 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:03.656374    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:55:11.478809 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:11.478778    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"]
Sep 11 16:55:26.497577 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.497520    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" podUID="2a80c96b-e2d2-4259-9e93-d0328685ceba" containerName="main" containerID="cri-o://f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9" gracePeriod=45
Sep 11 16:55:26.566944 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.566908    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" podUID="2a80c96b-e2d2-4259-9e93-d0328685ceba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Sep 11 16:55:26.745612 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.745584    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:55:26.835588 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835565    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a80c96b-e2d2-4259-9e93-d0328685ceba-tls-certs\") pod \"2a80c96b-e2d2-4259-9e93-d0328685ceba\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") "
Sep 11 16:55:26.835741 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835600    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-home\") pod \"2a80c96b-e2d2-4259-9e93-d0328685ceba\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") "
Sep 11 16:55:26.835741 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835647    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npt6g\" (UniqueName: \"kubernetes.io/projected/2a80c96b-e2d2-4259-9e93-d0328685ceba-kube-api-access-npt6g\") pod \"2a80c96b-e2d2-4259-9e93-d0328685ceba\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") "
Sep 11 16:55:26.835871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835802    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-model-cache\") pod \"2a80c96b-e2d2-4259-9e93-d0328685ceba\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") "
Sep 11 16:55:26.835871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835851    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-tmp-dir\") pod \"2a80c96b-e2d2-4259-9e93-d0328685ceba\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") "
Sep 11 16:55:26.835871 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835866    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-dshm\") pod \"2a80c96b-e2d2-4259-9e93-d0328685ceba\" (UID: \"2a80c96b-e2d2-4259-9e93-d0328685ceba\") "
Sep 11 16:55:26.836042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.835875    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-home" (OuterVolumeSpecName: "home") pod "2a80c96b-e2d2-4259-9e93-d0328685ceba" (UID: "2a80c96b-e2d2-4259-9e93-d0328685ceba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:26.836097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.836044    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a80c96b-e2d2-4259-9e93-d0328685ceba" (UID: "2a80c96b-e2d2-4259-9e93-d0328685ceba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:26.836097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.836054    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a80c96b-e2d2-4259-9e93-d0328685ceba" (UID: "2a80c96b-e2d2-4259-9e93-d0328685ceba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:26.836097 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.836093    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:26.837814 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.837773    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-dshm" (OuterVolumeSpecName: "dshm") pod "2a80c96b-e2d2-4259-9e93-d0328685ceba" (UID: "2a80c96b-e2d2-4259-9e93-d0328685ceba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:26.837914 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.837848    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a80c96b-e2d2-4259-9e93-d0328685ceba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a80c96b-e2d2-4259-9e93-d0328685ceba" (UID: "2a80c96b-e2d2-4259-9e93-d0328685ceba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:55:26.837914 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.837861    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a80c96b-e2d2-4259-9e93-d0328685ceba-kube-api-access-npt6g" (OuterVolumeSpecName: "kube-api-access-npt6g") pod "2a80c96b-e2d2-4259-9e93-d0328685ceba" (UID: "2a80c96b-e2d2-4259-9e93-d0328685ceba"). InnerVolumeSpecName "kube-api-access-npt6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:55:26.912570 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.912524    2550 generic.go:356] "Generic (PLEG): container finished" podID="2a80c96b-e2d2-4259-9e93-d0328685ceba" containerID="f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9" exitCode=0
Sep 11 16:55:26.912676 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.912588    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"
Sep 11 16:55:26.912676 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.912607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" event={"ID":"2a80c96b-e2d2-4259-9e93-d0328685ceba","Type":"ContainerDied","Data":"f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9"}
Sep 11 16:55:26.912676 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.912659    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc" event={"ID":"2a80c96b-e2d2-4259-9e93-d0328685ceba","Type":"ContainerDied","Data":"e434868cb9b5d6c6501f8e5830eef3283c2a7705c94d2e6ef026fd3c3dda81f9"}
Sep 11 16:55:26.912780 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.912680    2550 scope.go:117] "RemoveContainer" containerID="f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9"
Sep 11 16:55:26.922217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.922200    2550 scope.go:117] "RemoveContainer" containerID="f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9"
Sep 11 16:55:26.922506 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:55:26.922484    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9\": container with ID starting with f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9 not found: ID does not exist" containerID="f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9"
Sep 11 16:55:26.922580 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.922513    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9"} err="failed to get container status \"f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9\": rpc error: code = NotFound desc = could not find container \"f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9\": container with ID starting with f1f4fa1d9f4b0000c0d6a719082262c6e9685603f04cf4c69b362e99557cceb9 not found: ID does not exist"
Sep 11 16:55:26.933145 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.933120    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"]
Sep 11 16:55:26.936478 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.936458    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-lwdcc"]
Sep 11 16:55:26.936677 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.936661    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npt6g\" (UniqueName: \"kubernetes.io/projected/2a80c96b-e2d2-4259-9e93-d0328685ceba-kube-api-access-npt6g\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:26.936718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.936682    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:26.936718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.936693    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:26.936718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.936700    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a80c96b-e2d2-4259-9e93-d0328685ceba-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:26.936718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:26.936707    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a80c96b-e2d2-4259-9e93-d0328685ceba-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:27.477180 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:27.477152    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a80c96b-e2d2-4259-9e93-d0328685ceba" path="/var/lib/kubelet/pods/2a80c96b-e2d2-4259-9e93-d0328685ceba/volumes"
Sep 11 16:55:27.567891 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:27.567859    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"]
Sep 11 16:55:42.589881 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.589843    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh" podUID="e9cfa438-9f02-4369-83ab-83f89a074ddf" containerName="main" containerID="cri-o://4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71" gracePeriod=45
Sep 11 16:55:42.833846 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.833824    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:55:42.860603 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860527    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-dshm\") pod \"e9cfa438-9f02-4369-83ab-83f89a074ddf\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") "
Sep 11 16:55:42.860717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860610    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-model-cache\") pod \"e9cfa438-9f02-4369-83ab-83f89a074ddf\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") "
Sep 11 16:55:42.860717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860644    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-tmp-dir\") pod \"e9cfa438-9f02-4369-83ab-83f89a074ddf\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") "
Sep 11 16:55:42.860717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860685    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-home\") pod \"e9cfa438-9f02-4369-83ab-83f89a074ddf\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") "
Sep 11 16:55:42.860717 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860715    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cfa438-9f02-4369-83ab-83f89a074ddf-tls-certs\") pod \"e9cfa438-9f02-4369-83ab-83f89a074ddf\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") "
Sep 11 16:55:42.860950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860752    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgcds\" (UniqueName: \"kubernetes.io/projected/e9cfa438-9f02-4369-83ab-83f89a074ddf-kube-api-access-hgcds\") pod \"e9cfa438-9f02-4369-83ab-83f89a074ddf\" (UID: \"e9cfa438-9f02-4369-83ab-83f89a074ddf\") "
Sep 11 16:55:42.860950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860852    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9cfa438-9f02-4369-83ab-83f89a074ddf" (UID: "e9cfa438-9f02-4369-83ab-83f89a074ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:42.860950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860925    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-home" (OuterVolumeSpecName: "home") pod "e9cfa438-9f02-4369-83ab-83f89a074ddf" (UID: "e9cfa438-9f02-4369-83ab-83f89a074ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:42.860950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860933    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9cfa438-9f02-4369-83ab-83f89a074ddf" (UID: "e9cfa438-9f02-4369-83ab-83f89a074ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:42.861167 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.860985    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:42.861167 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.861025    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:42.861167 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.861038    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:42.862718 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.862696    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-dshm" (OuterVolumeSpecName: "dshm") pod "e9cfa438-9f02-4369-83ab-83f89a074ddf" (UID: "e9cfa438-9f02-4369-83ab-83f89a074ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:55:42.862836 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.862809    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cfa438-9f02-4369-83ab-83f89a074ddf-kube-api-access-hgcds" (OuterVolumeSpecName: "kube-api-access-hgcds") pod "e9cfa438-9f02-4369-83ab-83f89a074ddf" (UID: "e9cfa438-9f02-4369-83ab-83f89a074ddf"). InnerVolumeSpecName "kube-api-access-hgcds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:55:42.863196 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.863171    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9cfa438-9f02-4369-83ab-83f89a074ddf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9cfa438-9f02-4369-83ab-83f89a074ddf" (UID: "e9cfa438-9f02-4369-83ab-83f89a074ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:55:42.961902 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.961877    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cfa438-9f02-4369-83ab-83f89a074ddf-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:42.961902 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.961897    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cfa438-9f02-4369-83ab-83f89a074ddf-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:42.962045 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.961908    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgcds\" (UniqueName: \"kubernetes.io/projected/e9cfa438-9f02-4369-83ab-83f89a074ddf-kube-api-access-hgcds\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:55:42.993687 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.993656    2550 generic.go:356] "Generic (PLEG): container finished" podID="e9cfa438-9f02-4369-83ab-83f89a074ddf" containerID="4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71" exitCode=0
Sep 11 16:55:42.993791 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.993706    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"
Sep 11 16:55:42.993791 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.993737    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh" event={"ID":"e9cfa438-9f02-4369-83ab-83f89a074ddf","Type":"ContainerDied","Data":"4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71"}
Sep 11 16:55:42.993876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.993805    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh" event={"ID":"e9cfa438-9f02-4369-83ab-83f89a074ddf","Type":"ContainerDied","Data":"0c47fa9151dedfec5fd54d756a0f9d14fd5c216b060bde4d58fdcfd86b61371a"}
Sep 11 16:55:42.993876 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:42.993827    2550 scope.go:117] "RemoveContainer" containerID="4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71"
Sep 11 16:55:43.002922 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:43.002907    2550 scope.go:117] "RemoveContainer" containerID="4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71"
Sep 11 16:55:43.003189 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:55:43.003168    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71\": container with ID starting with 4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71 not found: ID does not exist" containerID="4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71"
Sep 11 16:55:43.003241 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:43.003197    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71"} err="failed to get container status \"4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71\": rpc error: code = NotFound desc = could not find container \"4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71\": container with ID starting with 4c072dc3ed388051f2a19b72759555944eacf4d188da3fe4bc85296a35473d71 not found: ID does not exist"
Sep 11 16:55:43.013699 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:43.013678    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"]
Sep 11 16:55:43.017063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:43.017043    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-xpwdh"]
Sep 11 16:55:43.477643 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:55:43.477610    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9cfa438-9f02-4369-83ab-83f89a074ddf" path="/var/lib/kubelet/pods/e9cfa438-9f02-4369-83ab-83f89a074ddf/volumes"
Sep 11 16:56:40.499334 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:56:40.499300    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"]
Sep 11 16:56:43.656635 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:56:43.656597    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:56:53.656555 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:56:53.656503    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:56:55.519140 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:56:55.519102    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main" containerID="cri-o://65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27" gracePeriod=45
Sep 11 16:57:03.656690 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:03.656653    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:57:03.657144 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:03.656747    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:57:05.616636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:05.616610    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:57:05.626654 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:05.626632    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 16:57:13.656468 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:13.656431    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:57:23.656733 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:23.656696    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:57:25.755063 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.755039    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:57:25.818618 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.818535    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8e4faa-2da6-4fef-a461-87693c9ae87a-tls-certs\") pod \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") "
Sep 11 16:57:25.818618 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.818597    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kclh\" (UniqueName: \"kubernetes.io/projected/ff8e4faa-2da6-4fef-a461-87693c9ae87a-kube-api-access-9kclh\") pod \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\" (UID: \"ff8e4faa-2da6-4fef-a461-87693c9ae87a\") "
Sep 11 16:57:25.820636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.820605    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff8e4faa-2da6-4fef-a461-87693c9ae87a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff8e4faa-2da6-4fef-a461-87693c9ae87a" (UID: "ff8e4faa-2da6-4fef-a461-87693c9ae87a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:57:25.820751 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.820667    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8e4faa-2da6-4fef-a461-87693c9ae87a-kube-api-access-9kclh" (OuterVolumeSpecName: "kube-api-access-9kclh") pod "ff8e4faa-2da6-4fef-a461-87693c9ae87a" (UID: "ff8e4faa-2da6-4fef-a461-87693c9ae87a"). InnerVolumeSpecName "kube-api-access-9kclh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:57:25.920024 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.919998    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff8e4faa-2da6-4fef-a461-87693c9ae87a-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:57:25.920024 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:25.920022    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kclh\" (UniqueName: \"kubernetes.io/projected/ff8e4faa-2da6-4fef-a461-87693c9ae87a-kube-api-access-9kclh\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:57:26.489027 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.488987    2550 generic.go:356] "Generic (PLEG): container finished" podID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerID="65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27" exitCode=0
Sep 11 16:57:26.489217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.489045    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"
Sep 11 16:57:26.489217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.489073    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" event={"ID":"ff8e4faa-2da6-4fef-a461-87693c9ae87a","Type":"ContainerDied","Data":"65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27"}
Sep 11 16:57:26.489217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.489123    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm" event={"ID":"ff8e4faa-2da6-4fef-a461-87693c9ae87a","Type":"ContainerDied","Data":"d2f5ed6c04b593c9797e37db3f29728adfc13721591e7ff902b228d535798ee0"}
Sep 11 16:57:26.489217 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.489145    2550 scope.go:117] "RemoveContainer" containerID="65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27"
Sep 11 16:57:26.497752 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.497737    2550 scope.go:117] "RemoveContainer" containerID="65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27"
Sep 11 16:57:26.498018 ip-10-0-141-84 kubenswrapper[2550]: E0911 16:57:26.498000    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27\": container with ID starting with 65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27 not found: ID does not exist" containerID="65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27"
Sep 11 16:57:26.498082 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.498025    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27"} err="failed to get container status \"65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27\": rpc error: code = NotFound desc = could not find container \"65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27\": container with ID starting with 65f3fb91218dad56455a5f43e24275dc34cbd54f719d98c799d3e04ae4197b27 not found: ID does not exist"
Sep 11 16:57:26.508108 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.508087    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"]
Sep 11 16:57:26.511983 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:26.511963    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pxnhm"]
Sep 11 16:57:27.476834 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:27.476800    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" path="/var/lib/kubelet/pods/ff8e4faa-2da6-4fef-a461-87693c9ae87a/volumes"
Sep 11 16:57:59.636182 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636150    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"]
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636458    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636468    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636480    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a80c96b-e2d2-4259-9e93-d0328685ceba" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636486    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a80c96b-e2d2-4259-9e93-d0328685ceba" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636498    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cfa438-9f02-4369-83ab-83f89a074ddf" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636505    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cfa438-9f02-4369-83ab-83f89a074ddf" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636585    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a80c96b-e2d2-4259-9e93-d0328685ceba" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636597    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cfa438-9f02-4369-83ab-83f89a074ddf" containerName="main"
Sep 11 16:57:59.638686 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.636605    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8e4faa-2da6-4fef-a461-87693c9ae87a" containerName="main"
Sep 11 16:57:59.639689 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.639662    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.641957 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.641934    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-bqx9c\""
Sep 11 16:57:59.642366 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.642341    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 11 16:57:59.642478 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.642396    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 11 16:57:59.642478 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.642428    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-9vp65\""
Sep 11 16:57:59.642642 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.642478    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 11 16:57:59.648772 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.648748    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"]
Sep 11 16:57:59.755372 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.755339    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffef180f-d9dd-4daf-80cf-f980f851ec50-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.755488 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.755439    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ct8\" (UniqueName: \"kubernetes.io/projected/ffef180f-d9dd-4daf-80cf-f980f851ec50-kube-api-access-v7ct8\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.856728 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.856699    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffef180f-d9dd-4daf-80cf-f980f851ec50-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.856858 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.856759    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ct8\" (UniqueName: \"kubernetes.io/projected/ffef180f-d9dd-4daf-80cf-f980f851ec50-kube-api-access-v7ct8\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.859179 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.859159    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffef180f-d9dd-4daf-80cf-f980f851ec50-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.864350 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.864328    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ct8\" (UniqueName: \"kubernetes.io/projected/ffef180f-d9dd-4daf-80cf-f980f851ec50-kube-api-access-v7ct8\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:57:59.950772 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:57:59.950713    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:58:00.222466 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:00.222438    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"]
Sep 11 16:58:00.224478 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:58:00.224452    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffef180f_d9dd_4daf_80cf_f980f851ec50.slice/crio-197c71655118187b850e8cf903065379088ae19183ef9866de9e11a39c35f13a WatchSource:0}: Error finding container 197c71655118187b850e8cf903065379088ae19183ef9866de9e11a39c35f13a: Status 404 returned error can't find the container with id 197c71655118187b850e8cf903065379088ae19183ef9866de9e11a39c35f13a
Sep 11 16:58:00.645450 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:00.645412    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" event={"ID":"ffef180f-d9dd-4daf-80cf-f980f851ec50","Type":"ContainerStarted","Data":"7acbcf3ebe3062af5f5eaa8dab232d91ba59e722ad42947de4fbef5e87125f2d"}
Sep 11 16:58:00.645861 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:00.645477    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" event={"ID":"ffef180f-d9dd-4daf-80cf-f980f851ec50","Type":"ContainerStarted","Data":"197c71655118187b850e8cf903065379088ae19183ef9866de9e11a39c35f13a"}
Sep 11 16:58:00.645861 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:00.645536    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:58:00.661930 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:00.661883    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" podStartSLOduration=1.6618702650000001 podStartE2EDuration="1.661870265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:00.661045572 +0000 UTC m=+2455.617093197" watchObservedRunningTime="2026-09-11 16:58:00.661870265 +0000 UTC m=+2455.617917892"
Sep 11 16:58:31.654930 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:31.654899    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:58:36.705427 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.705396    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"]
Sep 11 16:58:36.708575 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.708556    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.710814 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.710791    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-6tbhd\""
Sep 11 16:58:36.710950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.710853    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 11 16:58:36.710950 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.710882    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 11 16:58:36.711148 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.711134    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 11 16:58:36.718328 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.718308    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"]
Sep 11 16:58:36.735891 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.735865    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spqq9\" (UniqueName: \"kubernetes.io/projected/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-kube-api-access-spqq9\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.735994 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.735905    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-model-cache\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.735994 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.735929    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-home\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.736077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.735996    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-dshm\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.736077 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.736067    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.736143 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.736084    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tls-certs\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837204 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837178    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-dshm\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837356 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837233    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837356 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837346    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tls-certs\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837481 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837425    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spqq9\" (UniqueName: \"kubernetes.io/projected/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-kube-api-access-spqq9\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837481 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837464    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-model-cache\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837628 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837501    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-home\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837684 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837657    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837829 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837805    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-model-cache\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.837932 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.837829    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-home\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.839637 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.839616    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-dshm\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.839936 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.839917    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tls-certs\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:36.844377 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:36.844359    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spqq9\" (UniqueName: \"kubernetes.io/projected/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-kube-api-access-spqq9\") pod \"late-v1-kserve-56cbc8456c-p7d96\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:37.020607 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:37.020530    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:37.312767 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:37.312653    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"]
Sep 11 16:58:37.315531 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:58:37.315506    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbb30f1_c3a8_48d1_ba96_72abb66b45ac.slice/crio-a22f76654cf9dddd93180fb70375144cf0ed1df0a862b49bc64d10a4a92ea908 WatchSource:0}: Error finding container a22f76654cf9dddd93180fb70375144cf0ed1df0a862b49bc64d10a4a92ea908: Status 404 returned error can't find the container with id a22f76654cf9dddd93180fb70375144cf0ed1df0a862b49bc64d10a4a92ea908
Sep 11 16:58:37.828645 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:37.828605    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96" event={"ID":"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac","Type":"ContainerStarted","Data":"532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13"}
Sep 11 16:58:37.829042 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:37.828672    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96" event={"ID":"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac","Type":"ContainerStarted","Data":"a22f76654cf9dddd93180fb70375144cf0ed1df0a862b49bc64d10a4a92ea908"}
Sep 11 16:58:37.846823 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:37.846766    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96" podStartSLOduration=1.8467497449999999 podStartE2EDuration="1.846749745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:37.845005788 +0000 UTC m=+2492.801053419" watchObservedRunningTime="2026-09-11 16:58:37.846749745 +0000 UTC m=+2492.802797376"
Sep 11 16:58:38.020647 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:38.020604    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:42.828509 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:42.828473    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"]
Sep 11 16:58:47.021636 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:47.021608    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:47.034763 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:47.034738    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:47.889620 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:47.889593    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 16:58:51.655126 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:51.655082    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:58:57.848745 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:58:57.848684    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main" containerID="cri-o://7acbcf3ebe3062af5f5eaa8dab232d91ba59e722ad42947de4fbef5e87125f2d" gracePeriod=45
Sep 11 16:59:01.655356 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:01.655313    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:59:11.654853 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:11.654814    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:59:11.655348 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:11.654913    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:59:21.654877 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:21.654840    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:59:28.069683 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.069651    2550 generic.go:356] "Generic (PLEG): container finished" podID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerID="7acbcf3ebe3062af5f5eaa8dab232d91ba59e722ad42947de4fbef5e87125f2d" exitCode=0
Sep 11 16:59:28.070083 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.069696    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" event={"ID":"ffef180f-d9dd-4daf-80cf-f980f851ec50","Type":"ContainerDied","Data":"7acbcf3ebe3062af5f5eaa8dab232d91ba59e722ad42947de4fbef5e87125f2d"}
Sep 11 16:59:28.095109 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.095087    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:59:28.215848 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.215774    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffef180f-d9dd-4daf-80cf-f980f851ec50-tls-certs\") pod \"ffef180f-d9dd-4daf-80cf-f980f851ec50\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") "
Sep 11 16:59:28.215848 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.215841    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7ct8\" (UniqueName: \"kubernetes.io/projected/ffef180f-d9dd-4daf-80cf-f980f851ec50-kube-api-access-v7ct8\") pod \"ffef180f-d9dd-4daf-80cf-f980f851ec50\" (UID: \"ffef180f-d9dd-4daf-80cf-f980f851ec50\") "
Sep 11 16:59:28.218017 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.217982    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffef180f-d9dd-4daf-80cf-f980f851ec50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffef180f-d9dd-4daf-80cf-f980f851ec50" (UID: "ffef180f-d9dd-4daf-80cf-f980f851ec50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:59:28.218135 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.218033    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffef180f-d9dd-4daf-80cf-f980f851ec50-kube-api-access-v7ct8" (OuterVolumeSpecName: "kube-api-access-v7ct8") pod "ffef180f-d9dd-4daf-80cf-f980f851ec50" (UID: "ffef180f-d9dd-4daf-80cf-f980f851ec50"). InnerVolumeSpecName "kube-api-access-v7ct8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:59:28.316619 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.316591    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7ct8\" (UniqueName: \"kubernetes.io/projected/ffef180f-d9dd-4daf-80cf-f980f851ec50-kube-api-access-v7ct8\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:59:28.316619 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:28.316616    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffef180f-d9dd-4daf-80cf-f980f851ec50-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 16:59:29.076663 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:29.076630    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"
Sep 11 16:59:29.077090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:29.076632    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9" event={"ID":"ffef180f-d9dd-4daf-80cf-f980f851ec50","Type":"ContainerDied","Data":"197c71655118187b850e8cf903065379088ae19183ef9866de9e11a39c35f13a"}
Sep 11 16:59:29.077090 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:29.076770    2550 scope.go:117] "RemoveContainer" containerID="7acbcf3ebe3062af5f5eaa8dab232d91ba59e722ad42947de4fbef5e87125f2d"
Sep 11 16:59:29.097081 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:29.097059    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"]
Sep 11 16:59:29.102586 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:29.102565    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785llsn9"]
Sep 11 16:59:29.477487 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:29.477459    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" path="/var/lib/kubelet/pods/ffef180f-d9dd-4daf-80cf-f980f851ec50/volumes"
Sep 11 16:59:36.645672 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.645614    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"]
Sep 11 16:59:36.646246 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.646107    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main"
Sep 11 16:59:36.646246 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.646125    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main"
Sep 11 16:59:36.646246 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.646204    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffef180f-d9dd-4daf-80cf-f980f851ec50" containerName="main"
Sep 11 16:59:36.650673 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.650652    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.653253 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.653234    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-bhf9z\""
Sep 11 16:59:36.653253 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.653244    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 11 16:59:36.653479 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.653256    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 11 16:59:36.653479 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.653250    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 11 16:59:36.653479 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.653235    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-jcvlw\""
Sep 11 16:59:36.659016 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.658995    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"]
Sep 11 16:59:36.778716 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.778682    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5f9\" (UniqueName: \"kubernetes.io/projected/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-kube-api-access-vw5f9\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.778875 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.778789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.879404 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.879370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.879573 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.879439    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5f9\" (UniqueName: \"kubernetes.io/projected/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-kube-api-access-vw5f9\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.881917 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.881895    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.886583 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.886564    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5f9\" (UniqueName: \"kubernetes.io/projected/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-kube-api-access-vw5f9\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:36.961831 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:36.961780    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:37.238021 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:37.237998    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"]
Sep 11 16:59:37.240070 ip-10-0-141-84 kubenswrapper[2550]: W0911 16:59:37.240038    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f47483_99b5_4983_8f8b_b2e0ffd44ec4.slice/crio-c32723cea7143964f9afe7d28544fd37a2e49efd4ab2e090e3d0e4c0ec739d65 WatchSource:0}: Error finding container c32723cea7143964f9afe7d28544fd37a2e49efd4ab2e090e3d0e4c0ec739d65: Status 404 returned error can't find the container with id c32723cea7143964f9afe7d28544fd37a2e49efd4ab2e090e3d0e4c0ec739d65
Sep 11 16:59:38.121010 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:38.120976    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" event={"ID":"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4","Type":"ContainerStarted","Data":"4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4"}
Sep 11 16:59:38.121430 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:38.121031    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" event={"ID":"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4","Type":"ContainerStarted","Data":"c32723cea7143964f9afe7d28544fd37a2e49efd4ab2e090e3d0e4c0ec739d65"}
Sep 11 16:59:38.121430 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:38.121072    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 16:59:38.136288 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:38.136245    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podStartSLOduration=2.136231561 podStartE2EDuration="2.136231561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:38.134772519 +0000 UTC m=+2553.090820146" watchObservedRunningTime="2026-09-11 16:59:38.136231561 +0000 UTC m=+2553.092279189"
Sep 11 16:59:56.272787 ip-10-0-141-84 kubenswrapper[2550]: I0911 16:59:56.272746    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"]
Sep 11 17:00:09.129046 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:09.129014    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 17:00:11.292295 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.292233    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96" podUID="5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" containerName="main" containerID="cri-o://532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13" gracePeriod=45
Sep 11 17:00:11.533417 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.533392    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 17:00:11.536313 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536293    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-model-cache\") pod \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") "
Sep 11 17:00:11.536374 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536338    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spqq9\" (UniqueName: \"kubernetes.io/projected/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-kube-api-access-spqq9\") pod \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") "
Sep 11 17:00:11.536416 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536380    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tls-certs\") pod \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") "
Sep 11 17:00:11.536416 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536409    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-dshm\") pod \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") "
Sep 11 17:00:11.536519 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536433    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-home\") pod \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") "
Sep 11 17:00:11.536519 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536460    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tmp-dir\") pod \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\" (UID: \"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac\") "
Sep 11 17:00:11.536638 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536536    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" (UID: "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:11.536698 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536666    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:00:11.536754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536740    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-home" (OuterVolumeSpecName: "home") pod "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" (UID: "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:11.536817 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.536788    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" (UID: "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:11.538592 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.538568    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-dshm" (OuterVolumeSpecName: "dshm") pod "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" (UID: "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:11.538688 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.538653    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" (UID: "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:11.538688 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.538673    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-kube-api-access-spqq9" (OuterVolumeSpecName: "kube-api-access-spqq9") pod "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" (UID: "5cbb30f1-c3a8-48d1-ba96-72abb66b45ac"). InnerVolumeSpecName "kube-api-access-spqq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:00:11.637373 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.637338    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spqq9\" (UniqueName: \"kubernetes.io/projected/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-kube-api-access-spqq9\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:00:11.637373 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.637370    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:00:11.637583 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.637382    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:00:11.637583 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.637391    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:00:11.637583 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:11.637399    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:00:12.286304 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.286263    2550 generic.go:356] "Generic (PLEG): container finished" podID="5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" containerID="532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13" exitCode=0
Sep 11 17:00:12.286476 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.286308    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"
Sep 11 17:00:12.286476 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.286353    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96" event={"ID":"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac","Type":"ContainerDied","Data":"532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13"}
Sep 11 17:00:12.286476 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.286406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96" event={"ID":"5cbb30f1-c3a8-48d1-ba96-72abb66b45ac","Type":"ContainerDied","Data":"a22f76654cf9dddd93180fb70375144cf0ed1df0a862b49bc64d10a4a92ea908"}
Sep 11 17:00:12.286476 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.286430    2550 scope.go:117] "RemoveContainer" containerID="532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13"
Sep 11 17:00:12.295464 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.295264    2550 scope.go:117] "RemoveContainer" containerID="532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13"
Sep 11 17:00:12.295786 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:00:12.295576    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13\": container with ID starting with 532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13 not found: ID does not exist" containerID="532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13"
Sep 11 17:00:12.295786 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.295608    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13"} err="failed to get container status \"532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13\": rpc error: code = NotFound desc = could not find container \"532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13\": container with ID starting with 532d66c9d46ca4303e8d3368e43ea648d3aff098bd9f77adc706fab42cd46a13 not found: ID does not exist"
Sep 11 17:00:12.306920 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.306895    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"]
Sep 11 17:00:12.310422 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:12.310404    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-p7d96"]
Sep 11 17:00:13.476352 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:13.476318    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" path="/var/lib/kubelet/pods/5cbb30f1-c3a8-48d1-ba96-72abb66b45ac/volumes"
Sep 11 17:00:24.874370 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.874336    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"]
Sep 11 17:00:24.874969 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.874667    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" containerName="main"
Sep 11 17:00:24.874969 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.874679    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" containerName="main"
Sep 11 17:00:24.874969 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.874741    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cbb30f1-c3a8-48d1-ba96-72abb66b45ac" containerName="main"
Sep 11 17:00:24.877662 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.877642    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:24.880133 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.880107    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 11 17:00:24.880133 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.880122    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-v8pph\""
Sep 11 17:00:24.880306 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.880143    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 11 17:00:24.880306 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.880110    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 11 17:00:24.886504 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.886482    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"]
Sep 11 17:00:24.936791 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.936765    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce4665f-09f4-4516-a8fe-bcf92a21504b-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:24.936929 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.936808    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-home\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:24.936929 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.936836    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:24.937048 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.936920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65llt\" (UniqueName: \"kubernetes.io/projected/fce4665f-09f4-4516-a8fe-bcf92a21504b-kube-api-access-65llt\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:24.937048 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.936964    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:24.937048 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:24.937034    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038299 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-home\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038433 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038433 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038349    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65llt\" (UniqueName: \"kubernetes.io/projected/fce4665f-09f4-4516-a8fe-bcf92a21504b-kube-api-access-65llt\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038433 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038433 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038408    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038685 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038482    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce4665f-09f4-4516-a8fe-bcf92a21504b-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038918 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038696    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-home\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038918 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038806    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.038918 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.038904    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.040691 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.040671    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.040868 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.040852    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce4665f-09f4-4516-a8fe-bcf92a21504b-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.046098 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.046079    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65llt\" (UniqueName: \"kubernetes.io/projected/fce4665f-09f4-4516-a8fe-bcf92a21504b-kube-api-access-65llt\") pod \"delw-v2-kserve-5b84cc7ddf-vtmqk\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.190131 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.190064    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:25.466082 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:25.465995    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"]
Sep 11 17:00:25.468382 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:00:25.468350    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce4665f_09f4_4516_a8fe_bcf92a21504b.slice/crio-de219eeb209bdfe5c51313bee7549fa4ea5d23a15bc8f596319125e72fd3b0a1 WatchSource:0}: Error finding container de219eeb209bdfe5c51313bee7549fa4ea5d23a15bc8f596319125e72fd3b0a1: Status 404 returned error can't find the container with id de219eeb209bdfe5c51313bee7549fa4ea5d23a15bc8f596319125e72fd3b0a1
Sep 11 17:00:26.356813 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:26.356768    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk" event={"ID":"fce4665f-09f4-4516-a8fe-bcf92a21504b","Type":"ContainerStarted","Data":"76a314d2f29571d29aa222168c71289c914a421c8c4c97478b2cceecdeba4567"}
Sep 11 17:00:26.357248 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:26.356829    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk" event={"ID":"fce4665f-09f4-4516-a8fe-bcf92a21504b","Type":"ContainerStarted","Data":"de219eeb209bdfe5c51313bee7549fa4ea5d23a15bc8f596319125e72fd3b0a1"}
Sep 11 17:00:26.375746 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:26.375698    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk" podStartSLOduration=2.37568381 podStartE2EDuration="2.37568381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:00:26.373244899 +0000 UTC m=+2601.329292527" watchObservedRunningTime="2026-09-11 17:00:26.37568381 +0000 UTC m=+2601.331731437"
Sep 11 17:00:27.191173 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:27.191144    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:35.190558 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:35.190465    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:35.203315 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:35.203284    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:35.411107 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:35.411077    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:00:58.409340 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:58.409307    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"]
Sep 11 17:00:59.129443 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:00:59.129401    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:01:08.480401 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:08.480370    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"]
Sep 11 17:01:09.129624 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:09.129585    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:01:13.430136 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:13.430062    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main" containerID="cri-o://4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4" gracePeriod=45
Sep 11 17:01:19.129060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:19.129022    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:01:19.129476 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:19.129123    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 17:01:23.499247 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.499188    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk" podUID="fce4665f-09f4-4516-a8fe-bcf92a21504b" containerName="main" containerID="cri-o://76a314d2f29571d29aa222168c71289c914a421c8c4c97478b2cceecdeba4567" gracePeriod=45
Sep 11 17:01:23.633379 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.633351    2550 generic.go:356] "Generic (PLEG): container finished" podID="fce4665f-09f4-4516-a8fe-bcf92a21504b" containerID="76a314d2f29571d29aa222168c71289c914a421c8c4c97478b2cceecdeba4567" exitCode=0
Sep 11 17:01:23.633499 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.633440    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk" event={"ID":"fce4665f-09f4-4516-a8fe-bcf92a21504b","Type":"ContainerDied","Data":"76a314d2f29571d29aa222168c71289c914a421c8c4c97478b2cceecdeba4567"}
Sep 11 17:01:23.752572 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.752505    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:01:23.794398 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794370    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65llt\" (UniqueName: \"kubernetes.io/projected/fce4665f-09f4-4516-a8fe-bcf92a21504b-kube-api-access-65llt\") pod \"fce4665f-09f4-4516-a8fe-bcf92a21504b\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") "
Sep 11 17:01:23.794573 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794405    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-tmp-dir\") pod \"fce4665f-09f4-4516-a8fe-bcf92a21504b\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") "
Sep 11 17:01:23.794573 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794423    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-home\") pod \"fce4665f-09f4-4516-a8fe-bcf92a21504b\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") "
Sep 11 17:01:23.794573 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794460    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-model-cache\") pod \"fce4665f-09f4-4516-a8fe-bcf92a21504b\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") "
Sep 11 17:01:23.794760 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794594    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce4665f-09f4-4516-a8fe-bcf92a21504b-tls-certs\") pod \"fce4665f-09f4-4516-a8fe-bcf92a21504b\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") "
Sep 11 17:01:23.794760 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794628    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-dshm\") pod \"fce4665f-09f4-4516-a8fe-bcf92a21504b\" (UID: \"fce4665f-09f4-4516-a8fe-bcf92a21504b\") "
Sep 11 17:01:23.794760 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794685    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fce4665f-09f4-4516-a8fe-bcf92a21504b" (UID: "fce4665f-09f4-4516-a8fe-bcf92a21504b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:23.794760 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794711    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-model-cache" (OuterVolumeSpecName: "model-cache") pod "fce4665f-09f4-4516-a8fe-bcf92a21504b" (UID: "fce4665f-09f4-4516-a8fe-bcf92a21504b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:23.794760 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794704    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-home" (OuterVolumeSpecName: "home") pod "fce4665f-09f4-4516-a8fe-bcf92a21504b" (UID: "fce4665f-09f4-4516-a8fe-bcf92a21504b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:23.795015 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794876    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:23.795015 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794895    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:23.795015 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.794907    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:23.796754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.796723    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce4665f-09f4-4516-a8fe-bcf92a21504b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fce4665f-09f4-4516-a8fe-bcf92a21504b" (UID: "fce4665f-09f4-4516-a8fe-bcf92a21504b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:01:23.796754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.796742    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-dshm" (OuterVolumeSpecName: "dshm") pod "fce4665f-09f4-4516-a8fe-bcf92a21504b" (UID: "fce4665f-09f4-4516-a8fe-bcf92a21504b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:01:23.796865 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.796807    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce4665f-09f4-4516-a8fe-bcf92a21504b-kube-api-access-65llt" (OuterVolumeSpecName: "kube-api-access-65llt") pod "fce4665f-09f4-4516-a8fe-bcf92a21504b" (UID: "fce4665f-09f4-4516-a8fe-bcf92a21504b"). InnerVolumeSpecName "kube-api-access-65llt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:01:23.895672 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.895642    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65llt\" (UniqueName: \"kubernetes.io/projected/fce4665f-09f4-4516-a8fe-bcf92a21504b-kube-api-access-65llt\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:23.895672 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.895671    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce4665f-09f4-4516-a8fe-bcf92a21504b-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:23.895672 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:23.895681    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce4665f-09f4-4516-a8fe-bcf92a21504b-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:24.639635 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:24.639606    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"
Sep 11 17:01:24.639635 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:24.639633    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk" event={"ID":"fce4665f-09f4-4516-a8fe-bcf92a21504b","Type":"ContainerDied","Data":"de219eeb209bdfe5c51313bee7549fa4ea5d23a15bc8f596319125e72fd3b0a1"}
Sep 11 17:01:24.640184 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:24.639686    2550 scope.go:117] "RemoveContainer" containerID="76a314d2f29571d29aa222168c71289c914a421c8c4c97478b2cceecdeba4567"
Sep 11 17:01:24.661662 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:24.661638    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"]
Sep 11 17:01:24.665029 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:24.665005    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-vtmqk"]
Sep 11 17:01:25.479745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:25.479710    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce4665f-09f4-4516-a8fe-bcf92a21504b" path="/var/lib/kubelet/pods/fce4665f-09f4-4516-a8fe-bcf92a21504b/volumes"
Sep 11 17:01:29.129413 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:29.129375    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:01:39.129789 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:39.129751    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:01:43.678440 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.678411    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 17:01:43.735926 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.735856    2550 generic.go:356] "Generic (PLEG): container finished" podID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerID="4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4" exitCode=0
Sep 11 17:01:43.735926 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.735904    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"
Sep 11 17:01:43.735926 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.735918    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" event={"ID":"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4","Type":"ContainerDied","Data":"4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4"}
Sep 11 17:01:43.736136 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.735970    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj" event={"ID":"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4","Type":"ContainerDied","Data":"c32723cea7143964f9afe7d28544fd37a2e49efd4ab2e090e3d0e4c0ec739d65"}
Sep 11 17:01:43.736136 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.735999    2550 scope.go:117] "RemoveContainer" containerID="4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4"
Sep 11 17:01:43.738248 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.738216    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-tls-certs\") pod \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") "
Sep 11 17:01:43.738367 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.738296    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw5f9\" (UniqueName: \"kubernetes.io/projected/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-kube-api-access-vw5f9\") pod \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\" (UID: \"a5f47483-99b5-4983-8f8b-b2e0ffd44ec4\") "
Sep 11 17:01:43.740520 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.740495    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" (UID: "a5f47483-99b5-4983-8f8b-b2e0ffd44ec4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:01:43.740716 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.740697    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-kube-api-access-vw5f9" (OuterVolumeSpecName: "kube-api-access-vw5f9") pod "a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" (UID: "a5f47483-99b5-4983-8f8b-b2e0ffd44ec4"). InnerVolumeSpecName "kube-api-access-vw5f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:01:43.754701 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.754683    2550 scope.go:117] "RemoveContainer" containerID="4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4"
Sep 11 17:01:43.754952 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:01:43.754933    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4\": container with ID starting with 4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4 not found: ID does not exist" containerID="4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4"
Sep 11 17:01:43.755016 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.754959    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4"} err="failed to get container status \"4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4\": rpc error: code = NotFound desc = could not find container \"4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4\": container with ID starting with 4c6b099b079f1e51e632bb98a7f328145c95e71f31a3c134f766e2cf2905b8f4 not found: ID does not exist"
Sep 11 17:01:43.839107 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.839082    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5f9\" (UniqueName: \"kubernetes.io/projected/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-kube-api-access-vw5f9\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:43.839107 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:43.839103    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:01:44.056779 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:44.056745    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"]
Sep 11 17:01:44.058572 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:44.058552    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7drhczj"]
Sep 11 17:01:45.477122 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:01:45.477088    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" path="/var/lib/kubelet/pods/a5f47483-99b5-4983-8f8b-b2e0ffd44ec4/volumes"
Sep 11 17:02:05.646087 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:05.646018    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:02:05.655912 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:05.655888    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:02:06.080198 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080173    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"]
Sep 11 17:02:06.080484 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080473    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce4665f-09f4-4516-a8fe-bcf92a21504b" containerName="main"
Sep 11 17:02:06.080532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080486    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce4665f-09f4-4516-a8fe-bcf92a21504b" containerName="main"
Sep 11 17:02:06.080532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080496    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main"
Sep 11 17:02:06.080532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080501    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main"
Sep 11 17:02:06.080662 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080565    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f47483-99b5-4983-8f8b-b2e0ffd44ec4" containerName="main"
Sep 11 17:02:06.080662 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.080577    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce4665f-09f4-4516-a8fe-bcf92a21504b" containerName="main"
Sep 11 17:02:06.084976 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.084959    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.087302 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.087281    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 11 17:02:06.087302 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.087297    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-z4m6z\""
Sep 11 17:02:06.087302 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.087311    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 11 17:02:06.087517 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.087374    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 11 17:02:06.091233 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.091214    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"]
Sep 11 17:02:06.204076 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.204043    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.204076 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.204079    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.204272 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.204101    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.204272 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.204119    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rg2k\" (UniqueName: \"kubernetes.io/projected/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-kube-api-access-4rg2k\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.204272 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.204183    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.204272 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.204231    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.304989 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.304958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.304997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305030    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305053    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rg2k\" (UniqueName: \"kubernetes.io/projected/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-kube-api-access-4rg2k\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305081    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305130    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305443 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305416    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305442    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.305571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.305489    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.307351 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.307331    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.307568 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.307525    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.312039 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.312009    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rg2k\" (UniqueName: \"kubernetes.io/projected/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-kube-api-access-4rg2k\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.397163 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.397087    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:06.668445 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.668421    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"]
Sep 11 17:02:06.670522 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:02:06.670492    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf28f6b_15c6_4e06_ad31_81cf5a0d67d9.slice/crio-e8303fd5956d5e4ec2d210957e85b8d357d199f5616972a83e722c4103b7468d WatchSource:0}: Error finding container e8303fd5956d5e4ec2d210957e85b8d357d199f5616972a83e722c4103b7468d: Status 404 returned error can't find the container with id e8303fd5956d5e4ec2d210957e85b8d357d199f5616972a83e722c4103b7468d
Sep 11 17:02:06.847407 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.847373    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" event={"ID":"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9","Type":"ContainerStarted","Data":"8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"}
Sep 11 17:02:06.847591 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.847425    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" event={"ID":"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9","Type":"ContainerStarted","Data":"e8303fd5956d5e4ec2d210957e85b8d357d199f5616972a83e722c4103b7468d"}
Sep 11 17:02:06.862391 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:06.862346    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" podStartSLOduration=0.862331595 podStartE2EDuration="862.331595ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:02:06.861249992 +0000 UTC m=+2701.817297621" watchObservedRunningTime="2026-09-11 17:02:06.862331595 +0000 UTC m=+2701.818379271"
Sep 11 17:02:07.397392 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:07.397358    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:16.398150 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:16.398103    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:16.410401 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:16.410379    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:16.906240 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:16.906212    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:39.832446 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:39.832412    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"]
Sep 11 17:02:39.841278 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:02:39.841244    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:02:39.841278 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:02:39.841278 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:02:39.841278 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae" cmd=["/bin/sleep","15"]
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:39.841299    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" message="" err=<
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:02:39.841314    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerName="main" containerID="cri-o://8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"
Sep 11 17:02:39.841606 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:39.841370    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerName="main" containerID="cri-o://8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae" gracePeriod=60
Sep 11 17:02:39.895598 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:39.895564    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 11 17:02:40.189813 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.189790    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:40.361768 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361691    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-model-cache\") pod \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") "
Sep 11 17:02:40.361768 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361741    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tmp-dir\") pod \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") "
Sep 11 17:02:40.361768 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361767    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tls-certs\") pod \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") "
Sep 11 17:02:40.362024 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361797    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rg2k\" (UniqueName: \"kubernetes.io/projected/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-kube-api-access-4rg2k\") pod \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") "
Sep 11 17:02:40.362024 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361825    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-dshm\") pod \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") "
Sep 11 17:02:40.362024 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361884    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-home\") pod \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\" (UID: \"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9\") "
Sep 11 17:02:40.362024 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.361963    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" (UID: "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:02:40.362246 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.362050    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" (UID: "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:02:40.362246 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.362095    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:02:40.362246 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.362173    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-home" (OuterVolumeSpecName: "home") pod "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" (UID: "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:02:40.364013 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.363982    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-kube-api-access-4rg2k" (OuterVolumeSpecName: "kube-api-access-4rg2k") pod "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" (UID: "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9"). InnerVolumeSpecName "kube-api-access-4rg2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:02:40.364387 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.364359    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" (UID: "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:02:40.364493 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.364362    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-dshm" (OuterVolumeSpecName: "dshm") pod "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" (UID: "2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:02:40.462996 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.462968    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:02:40.462996 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.462995    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:02:40.463152 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.463008    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:02:40.463152 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.463020    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rg2k\" (UniqueName: \"kubernetes.io/projected/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-kube-api-access-4rg2k\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:02:40.463152 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:40.463030    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:02:41.008347 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.008313    2550 generic.go:356] "Generic (PLEG): container finished" podID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerID="8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae" exitCode=0
Sep 11 17:02:41.008777 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.008359    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"
Sep 11 17:02:41.008777 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.008403    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" event={"ID":"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9","Type":"ContainerDied","Data":"8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"}
Sep 11 17:02:41.008777 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.008457    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw" event={"ID":"2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9","Type":"ContainerDied","Data":"e8303fd5956d5e4ec2d210957e85b8d357d199f5616972a83e722c4103b7468d"}
Sep 11 17:02:41.008777 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.008490    2550 scope.go:117] "RemoveContainer" containerID="8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"
Sep 11 17:02:41.017995 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.017962    2550 scope.go:117] "RemoveContainer" containerID="8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"
Sep 11 17:02:41.018283 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:02:41.018264    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae\": container with ID starting with 8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae not found: ID does not exist" containerID="8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"
Sep 11 17:02:41.018347 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.018292    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae"} err="failed to get container status \"8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae\": rpc error: code = NotFound desc = could not find container \"8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae\": container with ID starting with 8511d2b5d6b47881c38cfe18ee0bf9c54644a0ab741fc27ed2a0e4c71668e5ae not found: ID does not exist"
Sep 11 17:02:41.028737 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.028712    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"]
Sep 11 17:02:41.030484 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.030462    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-fbxkw"]
Sep 11 17:02:41.478144 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:02:41.478101    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" path="/var/lib/kubelet/pods/2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9/volumes"
Sep 11 17:03:07.880935 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.880902    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"]
Sep 11 17:03:07.881435 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.881341    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerName="main"
Sep 11 17:03:07.881435 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.881359    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerName="main"
Sep 11 17:03:07.881587 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.881444    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf28f6b-15c6-4e06-ad31-81cf5a0d67d9" containerName="main"
Sep 11 17:03:07.884681 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.884662    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:07.886835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.886814    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 11 17:03:07.887391 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.887374    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-l8zg5\""
Sep 11 17:03:07.887472 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.887411    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 11 17:03:07.887472 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.887457    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 11 17:03:07.895006 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.894986    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"]
Sep 11 17:03:07.946817 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.946789    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:07.946942 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.946823    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-dshm\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:07.946942 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.946846    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-model-cache\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:07.946942 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.946872    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-home\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:07.947116 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.946942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bde41e-072a-44b5-b44c-719831bb8219-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:07.947116 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:07.947008    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmnkc\" (UniqueName: \"kubernetes.io/projected/18bde41e-072a-44b5-b44c-719831bb8219-kube-api-access-hmnkc\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047501 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047476    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047657 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047511    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-dshm\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047657 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047527    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-model-cache\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047657 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047591    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-home\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047657 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047633    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bde41e-072a-44b5-b44c-719831bb8219-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047876 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047676    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmnkc\" (UniqueName: \"kubernetes.io/projected/18bde41e-072a-44b5-b44c-719831bb8219-kube-api-access-hmnkc\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.047974 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047949    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.048036 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.047967    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-model-cache\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.048100 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.048049    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-home\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.049914 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.049886    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-dshm\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.050083 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.050054    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bde41e-072a-44b5-b44c-719831bb8219-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.054720 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.054701    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmnkc\" (UniqueName: \"kubernetes.io/projected/18bde41e-072a-44b5-b44c-719831bb8219-kube-api-access-hmnkc\") pod \"owner-v1-kserve-75579ff5bc-gj8d9\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.195442 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.195364    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:08.467727 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:08.467701    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"]
Sep 11 17:03:08.469329 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:03:08.469297    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bde41e_072a_44b5_b44c_719831bb8219.slice/crio-033e3816355305b38b0b40845b80562cff3325afc22d6f898d21a29634633ead WatchSource:0}: Error finding container 033e3816355305b38b0b40845b80562cff3325afc22d6f898d21a29634633ead: Status 404 returned error can't find the container with id 033e3816355305b38b0b40845b80562cff3325afc22d6f898d21a29634633ead
Sep 11 17:03:09.139467 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:09.139429    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9" event={"ID":"18bde41e-072a-44b5-b44c-719831bb8219","Type":"ContainerStarted","Data":"0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400"}
Sep 11 17:03:09.139870 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:09.139492    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9" event={"ID":"18bde41e-072a-44b5-b44c-719831bb8219","Type":"ContainerStarted","Data":"033e3816355305b38b0b40845b80562cff3325afc22d6f898d21a29634633ead"}
Sep 11 17:03:09.156421 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:09.156372    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9" podStartSLOduration=2.15635696 podStartE2EDuration="2.15635696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:09.153598479 +0000 UTC m=+2764.109646106" watchObservedRunningTime="2026-09-11 17:03:09.15635696 +0000 UTC m=+2764.112404618"
Sep 11 17:03:09.195626 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:09.195591    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:18.196017 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:18.195969    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:18.208321 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:18.208295    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:19.197417 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:19.197386    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:03:22.374568 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.374514    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"]
Sep 11 17:03:22.379222 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.379197    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.381194 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.381169    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 11 17:03:22.388457 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.388432    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"]
Sep 11 17:03:22.448868 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.448845    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.449002 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.448891    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-dshm\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.449002 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.448945    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.449002 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.448989    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-home\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.449120 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.449006    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a317d6d-9e57-4a12-a862-6ba8140b1570-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.449120 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.449027    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29r4f\" (UniqueName: \"kubernetes.io/projected/6a317d6d-9e57-4a12-a862-6ba8140b1570-kube-api-access-29r4f\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550246 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550217    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-home\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550393 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550251    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a317d6d-9e57-4a12-a862-6ba8140b1570-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550393 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550287    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29r4f\" (UniqueName: \"kubernetes.io/projected/6a317d6d-9e57-4a12-a862-6ba8140b1570-kube-api-access-29r4f\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550393 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550332    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550599 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550424    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-dshm\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550707 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550684    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550775 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550710    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-home\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.550775 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.550763    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.551022 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.551002    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-model-cache\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.552761 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.552734    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-dshm\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.553012 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.552990    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a317d6d-9e57-4a12-a862-6ba8140b1570-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.558220 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.558201    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29r4f\" (UniqueName: \"kubernetes.io/projected/6a317d6d-9e57-4a12-a862-6ba8140b1570-kube-api-access-29r4f\") pod \"owner-v2-kserve-5cf8959f95-mb8sx\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.691893 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.691823    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:22.961763 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:22.961688    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"]
Sep 11 17:03:22.963976 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:03:22.963945    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a317d6d_9e57_4a12_a862_6ba8140b1570.slice/crio-efda82fba6d381a54609939e9bc03c8ca8990f2ac5d2a080a86ed823dcf3cbe1 WatchSource:0}: Error finding container efda82fba6d381a54609939e9bc03c8ca8990f2ac5d2a080a86ed823dcf3cbe1: Status 404 returned error can't find the container with id efda82fba6d381a54609939e9bc03c8ca8990f2ac5d2a080a86ed823dcf3cbe1
Sep 11 17:03:23.205440 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:23.205406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx" event={"ID":"6a317d6d-9e57-4a12-a862-6ba8140b1570","Type":"ContainerStarted","Data":"73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b"}
Sep 11 17:03:23.205641 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:23.205459    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx" event={"ID":"6a317d6d-9e57-4a12-a862-6ba8140b1570","Type":"ContainerStarted","Data":"efda82fba6d381a54609939e9bc03c8ca8990f2ac5d2a080a86ed823dcf3cbe1"}
Sep 11 17:03:23.224507 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:23.224420    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx" podStartSLOduration=1.224403744 podStartE2EDuration="1.224403744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:23.222389944 +0000 UTC m=+2778.178437572" watchObservedRunningTime="2026-09-11 17:03:23.224403744 +0000 UTC m=+2778.180451373"
Sep 11 17:03:23.692658 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:23.692624    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:32.692516 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:32.692439    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:32.705838 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:32.705810    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:33.267131 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:33.267104    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:03:55.977746 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.977714    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"]
Sep 11 17:03:55.981112 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.981090    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:55.982945 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.982926    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 11 17:03:55.983290 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.983275    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 11 17:03:55.983381 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.983340    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-ptrfr\""
Sep 11 17:03:55.983445 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.983380    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 11 17:03:55.991985 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:55.991960    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"]
Sep 11 17:03:56.089009 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.088978    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6rs\" (UniqueName: \"kubernetes.io/projected/963c03d6-7f17-41ed-a02a-f967481311ba-kube-api-access-hx6rs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.089135 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.089026    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.089135 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.089045    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.089135 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.089075    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963c03d6-7f17-41ed-a02a-f967481311ba-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.089135 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.089111    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.089321 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.089143    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190324 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190296    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190330    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190359    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963c03d6-7f17-41ed-a02a-f967481311ba-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190384    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190404    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190756 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190583    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6rs\" (UniqueName: \"kubernetes.io/projected/963c03d6-7f17-41ed-a02a-f967481311ba-kube-api-access-hx6rs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.190853 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190827    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.191019 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.190915    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.191135 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.191111    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.192595 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.192571    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.193070 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.193052    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963c03d6-7f17-41ed-a02a-f967481311ba-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.204340 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.204314    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6rs\" (UniqueName: \"kubernetes.io/projected/963c03d6-7f17-41ed-a02a-f967481311ba-kube-api-access-hx6rs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.291966 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.291877    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:03:56.567981 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:56.567956    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"]
Sep 11 17:03:56.570256 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:03:56.570227    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963c03d6_7f17_41ed_a02a_f967481311ba.slice/crio-9d4dffb5a5a0e1ab49af8476800bfd8f1a6e65c598ab2b853ce95274ec3c562c WatchSource:0}: Error finding container 9d4dffb5a5a0e1ab49af8476800bfd8f1a6e65c598ab2b853ce95274ec3c562c: Status 404 returned error can't find the container with id 9d4dffb5a5a0e1ab49af8476800bfd8f1a6e65c598ab2b853ce95274ec3c562c
Sep 11 17:03:57.375833 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:57.375791    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" event={"ID":"963c03d6-7f17-41ed-a02a-f967481311ba","Type":"ContainerStarted","Data":"7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"}
Sep 11 17:03:57.376238 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:57.375854    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" event={"ID":"963c03d6-7f17-41ed-a02a-f967481311ba","Type":"ContainerStarted","Data":"9d4dffb5a5a0e1ab49af8476800bfd8f1a6e65c598ab2b853ce95274ec3c562c"}
Sep 11 17:03:57.394635 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:57.394593    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" podStartSLOduration=2.39457651 podStartE2EDuration="2.39457651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:03:57.392073676 +0000 UTC m=+2812.348121303" watchObservedRunningTime="2026-09-11 17:03:57.39457651 +0000 UTC m=+2812.350624135"
Sep 11 17:03:58.292708 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:03:58.292674    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:04:06.292298 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:06.292261    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:04:06.304846 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:06.304819    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:04:06.430980 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:06.430951    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:04:23.745852 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:23.745813    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"]
Sep 11 17:04:38.766284 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:38.766222    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9" podUID="18bde41e-072a-44b5-b44c-719831bb8219" containerName="main" containerID="cri-o://0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400" gracePeriod=45
Sep 11 17:04:39.013761 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.013728    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:04:39.117941 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.117911    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-model-cache\") pod \"18bde41e-072a-44b5-b44c-719831bb8219\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") "
Sep 11 17:04:39.118106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.117949    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-dshm\") pod \"18bde41e-072a-44b5-b44c-719831bb8219\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") "
Sep 11 17:04:39.118106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118016    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bde41e-072a-44b5-b44c-719831bb8219-tls-certs\") pod \"18bde41e-072a-44b5-b44c-719831bb8219\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") "
Sep 11 17:04:39.118106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118034    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmnkc\" (UniqueName: \"kubernetes.io/projected/18bde41e-072a-44b5-b44c-719831bb8219-kube-api-access-hmnkc\") pod \"18bde41e-072a-44b5-b44c-719831bb8219\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") "
Sep 11 17:04:39.118106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118056    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-home\") pod \"18bde41e-072a-44b5-b44c-719831bb8219\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") "
Sep 11 17:04:39.118334 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118112    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-tmp-dir\") pod \"18bde41e-072a-44b5-b44c-719831bb8219\" (UID: \"18bde41e-072a-44b5-b44c-719831bb8219\") "
Sep 11 17:04:39.118334 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118211    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-model-cache" (OuterVolumeSpecName: "model-cache") pod "18bde41e-072a-44b5-b44c-719831bb8219" (UID: "18bde41e-072a-44b5-b44c-719831bb8219"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:04:39.118435 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118380    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-home" (OuterVolumeSpecName: "home") pod "18bde41e-072a-44b5-b44c-719831bb8219" (UID: "18bde41e-072a-44b5-b44c-719831bb8219"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:04:39.118435 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118401    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:04:39.118435 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.118422    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18bde41e-072a-44b5-b44c-719831bb8219" (UID: "18bde41e-072a-44b5-b44c-719831bb8219"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:04:39.120222 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.120197    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bde41e-072a-44b5-b44c-719831bb8219-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18bde41e-072a-44b5-b44c-719831bb8219" (UID: "18bde41e-072a-44b5-b44c-719831bb8219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:04:39.120606 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.120584    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bde41e-072a-44b5-b44c-719831bb8219-kube-api-access-hmnkc" (OuterVolumeSpecName: "kube-api-access-hmnkc") pod "18bde41e-072a-44b5-b44c-719831bb8219" (UID: "18bde41e-072a-44b5-b44c-719831bb8219"). InnerVolumeSpecName "kube-api-access-hmnkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:04:39.120657 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.120633    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-dshm" (OuterVolumeSpecName: "dshm") pod "18bde41e-072a-44b5-b44c-719831bb8219" (UID: "18bde41e-072a-44b5-b44c-719831bb8219"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:04:39.218764 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.218740    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:04:39.218764 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.218761    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:04:39.218913 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.218769    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bde41e-072a-44b5-b44c-719831bb8219-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:04:39.218913 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.218779    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmnkc\" (UniqueName: \"kubernetes.io/projected/18bde41e-072a-44b5-b44c-719831bb8219-kube-api-access-hmnkc\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:04:39.218913 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.218788    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bde41e-072a-44b5-b44c-719831bb8219-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:04:39.586953 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.586923    2550 generic.go:356] "Generic (PLEG): container finished" podID="18bde41e-072a-44b5-b44c-719831bb8219" containerID="0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400" exitCode=0
Sep 11 17:04:39.587111 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.586966    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"
Sep 11 17:04:39.587111 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.587000    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9" event={"ID":"18bde41e-072a-44b5-b44c-719831bb8219","Type":"ContainerDied","Data":"0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400"}
Sep 11 17:04:39.587111 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.587054    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9" event={"ID":"18bde41e-072a-44b5-b44c-719831bb8219","Type":"ContainerDied","Data":"033e3816355305b38b0b40845b80562cff3325afc22d6f898d21a29634633ead"}
Sep 11 17:04:39.587111 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.587077    2550 scope.go:117] "RemoveContainer" containerID="0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400"
Sep 11 17:04:39.595745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.595726    2550 scope.go:117] "RemoveContainer" containerID="0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400"
Sep 11 17:04:39.596037 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:04:39.596013    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400\": container with ID starting with 0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400 not found: ID does not exist" containerID="0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400"
Sep 11 17:04:39.596086 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.596046    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400"} err="failed to get container status \"0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400\": rpc error: code = NotFound desc = could not find container \"0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400\": container with ID starting with 0b5471714e4fa2de474a8680b34461db35b292cfd323afb38f723e2421ba5400 not found: ID does not exist"
Sep 11 17:04:39.601287 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.601262    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"]
Sep 11 17:04:39.606746 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:39.606722    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-gj8d9"]
Sep 11 17:04:41.477213 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:41.477177    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18bde41e-072a-44b5-b44c-719831bb8219" path="/var/lib/kubelet/pods/18bde41e-072a-44b5-b44c-719831bb8219/volumes"
Sep 11 17:04:50.358424 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:50.358387    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"]
Sep 11 17:04:57.956731 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.956694    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"]
Sep 11 17:04:57.957247 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.957221    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bde41e-072a-44b5-b44c-719831bb8219" containerName="main"
Sep 11 17:04:57.957247 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.957247    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bde41e-072a-44b5-b44c-719831bb8219" containerName="main"
Sep 11 17:04:57.957451 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.957334    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bde41e-072a-44b5-b44c-719831bb8219" containerName="main"
Sep 11 17:04:57.960430 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.960406    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:57.962764 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.962737    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-kphjl\""
Sep 11 17:04:57.962873 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.962765    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 11 17:04:57.968813 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:57.968792    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"]
Sep 11 17:04:58.064776 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.064694    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83693c4-da71-4241-bb1b-1276c22d6c26-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.064776 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.064741    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvzn\" (UniqueName: \"kubernetes.io/projected/d83693c4-da71-4241-bb1b-1276c22d6c26-kube-api-access-fhvzn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.165931 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.165899    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83693c4-da71-4241-bb1b-1276c22d6c26-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.166107 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.165943    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvzn\" (UniqueName: \"kubernetes.io/projected/d83693c4-da71-4241-bb1b-1276c22d6c26-kube-api-access-fhvzn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.168374 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.168348    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83693c4-da71-4241-bb1b-1276c22d6c26-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.172644 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.172623    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvzn\" (UniqueName: \"kubernetes.io/projected/d83693c4-da71-4241-bb1b-1276c22d6c26-kube-api-access-fhvzn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.271616 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.271556    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.542671 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.542644    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"]
Sep 11 17:04:58.544786 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:04:58.544757    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83693c4_da71_4241_bb1b_1276c22d6c26.slice/crio-d0fbb68bbd4b5fd3d8f27ad357eb9fd5c9009f251f899d6c0d552715d55633fc WatchSource:0}: Error finding container d0fbb68bbd4b5fd3d8f27ad357eb9fd5c9009f251f899d6c0d552715d55633fc: Status 404 returned error can't find the container with id d0fbb68bbd4b5fd3d8f27ad357eb9fd5c9009f251f899d6c0d552715d55633fc
Sep 11 17:04:58.681146 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.681112    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" event={"ID":"d83693c4-da71-4241-bb1b-1276c22d6c26","Type":"ContainerStarted","Data":"8d42c9dd6792892110882cdf2599efac46de6abdda7879c715aef0fb01878920"}
Sep 11 17:04:58.681331 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.681168    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" event={"ID":"d83693c4-da71-4241-bb1b-1276c22d6c26","Type":"ContainerStarted","Data":"d0fbb68bbd4b5fd3d8f27ad357eb9fd5c9009f251f899d6c0d552715d55633fc"}
Sep 11 17:04:58.681331 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.681194    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:04:58.696595 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:04:58.696533    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" podStartSLOduration=1.6965178440000002 podStartE2EDuration="1.696517844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:58.69447614 +0000 UTC m=+2873.650523768" watchObservedRunningTime="2026-09-11 17:04:58.696517844 +0000 UTC m=+2873.652565471"
Sep 11 17:05:05.377081 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.377016    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx" podUID="6a317d6d-9e57-4a12-a862-6ba8140b1570" containerName="main" containerID="cri-o://73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b" gracePeriod=45
Sep 11 17:05:05.623701 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.623675    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:05:05.717680 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.717599    2550 generic.go:356] "Generic (PLEG): container finished" podID="6a317d6d-9e57-4a12-a862-6ba8140b1570" containerID="73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b" exitCode=0
Sep 11 17:05:05.717680 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.717645    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"
Sep 11 17:05:05.717680 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.717647    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx" event={"ID":"6a317d6d-9e57-4a12-a862-6ba8140b1570","Type":"ContainerDied","Data":"73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b"}
Sep 11 17:05:05.717902 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.717698    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx" event={"ID":"6a317d6d-9e57-4a12-a862-6ba8140b1570","Type":"ContainerDied","Data":"efda82fba6d381a54609939e9bc03c8ca8990f2ac5d2a080a86ed823dcf3cbe1"}
Sep 11 17:05:05.717902 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.717729    2550 scope.go:117] "RemoveContainer" containerID="73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b"
Sep 11 17:05:05.725978 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.725960    2550 scope.go:117] "RemoveContainer" containerID="73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b"
Sep 11 17:05:05.726273 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:05:05.726252    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b\": container with ID starting with 73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b not found: ID does not exist" containerID="73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b"
Sep 11 17:05:05.726342 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.726283    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b"} err="failed to get container status \"73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b\": rpc error: code = NotFound desc = could not find container \"73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b\": container with ID starting with 73b74f40e210217c8a28322fe77c276588ff5fc12965d2d7c712197f5848300b not found: ID does not exist"
Sep 11 17:05:05.727450 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727435    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-tmp-dir\") pod \"6a317d6d-9e57-4a12-a862-6ba8140b1570\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") "
Sep 11 17:05:05.727517 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727506    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-dshm\") pod \"6a317d6d-9e57-4a12-a862-6ba8140b1570\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") "
Sep 11 17:05:05.727567 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727529    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-model-cache\") pod \"6a317d6d-9e57-4a12-a862-6ba8140b1570\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") "
Sep 11 17:05:05.727610 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727589    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29r4f\" (UniqueName: \"kubernetes.io/projected/6a317d6d-9e57-4a12-a862-6ba8140b1570-kube-api-access-29r4f\") pod \"6a317d6d-9e57-4a12-a862-6ba8140b1570\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") "
Sep 11 17:05:05.727610 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727605    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a317d6d-9e57-4a12-a862-6ba8140b1570-tls-certs\") pod \"6a317d6d-9e57-4a12-a862-6ba8140b1570\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") "
Sep 11 17:05:05.727737 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727631    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-home\") pod \"6a317d6d-9e57-4a12-a862-6ba8140b1570\" (UID: \"6a317d6d-9e57-4a12-a862-6ba8140b1570\") "
Sep 11 17:05:05.727737 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727700    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a317d6d-9e57-4a12-a862-6ba8140b1570" (UID: "6a317d6d-9e57-4a12-a862-6ba8140b1570"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:05.727825 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.727796    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a317d6d-9e57-4a12-a862-6ba8140b1570" (UID: "6a317d6d-9e57-4a12-a862-6ba8140b1570"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:05.728410 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.728382    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-home" (OuterVolumeSpecName: "home") pod "6a317d6d-9e57-4a12-a862-6ba8140b1570" (UID: "6a317d6d-9e57-4a12-a862-6ba8140b1570"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:05.728569 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.728433    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:05:05.728697 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.728681    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:05:05.730474 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.730441    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-dshm" (OuterVolumeSpecName: "dshm") pod "6a317d6d-9e57-4a12-a862-6ba8140b1570" (UID: "6a317d6d-9e57-4a12-a862-6ba8140b1570"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:05.730607 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.730451    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a317d6d-9e57-4a12-a862-6ba8140b1570-kube-api-access-29r4f" (OuterVolumeSpecName: "kube-api-access-29r4f") pod "6a317d6d-9e57-4a12-a862-6ba8140b1570" (UID: "6a317d6d-9e57-4a12-a862-6ba8140b1570"). InnerVolumeSpecName "kube-api-access-29r4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:05:05.735310 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.735280    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a317d6d-9e57-4a12-a862-6ba8140b1570-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a317d6d-9e57-4a12-a862-6ba8140b1570" (UID: "6a317d6d-9e57-4a12-a862-6ba8140b1570"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:05:05.829491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.829467    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:05:05.829602 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.829491    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29r4f\" (UniqueName: \"kubernetes.io/projected/6a317d6d-9e57-4a12-a862-6ba8140b1570-kube-api-access-29r4f\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:05:05.829602 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.829506    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a317d6d-9e57-4a12-a862-6ba8140b1570-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:05:05.829602 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:05.829519    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a317d6d-9e57-4a12-a862-6ba8140b1570-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:05:06.037223 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:06.037188    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"]
Sep 11 17:05:06.039106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:06.039085    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-mb8sx"]
Sep 11 17:05:07.477003 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:07.476969    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a317d6d-9e57-4a12-a862-6ba8140b1570" path="/var/lib/kubelet/pods/6a317d6d-9e57-4a12-a862-6ba8140b1570/volumes"
Sep 11 17:05:29.687824 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:29.687795    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:05:50.910664 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:50.910625    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"]
Sep 11 17:05:59.688848 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:05:59.688808    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:05.932117 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:05.932072    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main" containerID="cri-o://8d42c9dd6792892110882cdf2599efac46de6abdda7879c715aef0fb01878920" gracePeriod=45
Sep 11 17:06:07.860512 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.860477    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"]
Sep 11 17:06:07.860890 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.860813    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a317d6d-9e57-4a12-a862-6ba8140b1570" containerName="main"
Sep 11 17:06:07.860890 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.860825    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a317d6d-9e57-4a12-a862-6ba8140b1570" containerName="main"
Sep 11 17:06:07.860967 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.860892    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a317d6d-9e57-4a12-a862-6ba8140b1570" containerName="main"
Sep 11 17:06:07.863755 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.863737    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:07.865882 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.865853    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 11 17:06:07.866030 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.865857    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 11 17:06:07.866318 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.866293    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 11 17:06:07.866444 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.866353    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-jz464\""
Sep 11 17:06:07.866499 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.866448    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-2dn59\""
Sep 11 17:06:07.874269 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.874247    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"]
Sep 11 17:06:07.885245 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.885221    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:07.885348 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.885270    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69qb\" (UniqueName: \"kubernetes.io/projected/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-kube-api-access-r69qb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:07.985877 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.985842    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69qb\" (UniqueName: \"kubernetes.io/projected/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-kube-api-access-r69qb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:07.986025 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.985939    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:07.988365 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.988341    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:07.993404 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:07.993378    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69qb\" (UniqueName: \"kubernetes.io/projected/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-kube-api-access-r69qb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:08.180823 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:08.180740    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:08.452402 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:08.452314    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"]
Sep 11 17:06:08.454990 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:06:08.454950    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc922fbdf_c5ce_49b2_93ad_c60e7031c5d3.slice/crio-df784d271e0888342d38338db55b29a6ba80331985ed4f5807c34d8ed318b3c1 WatchSource:0}: Error finding container df784d271e0888342d38338db55b29a6ba80331985ed4f5807c34d8ed318b3c1: Status 404 returned error can't find the container with id df784d271e0888342d38338db55b29a6ba80331985ed4f5807c34d8ed318b3c1
Sep 11 17:06:09.019857 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:09.019816    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" event={"ID":"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3","Type":"ContainerStarted","Data":"1702871fec3605e1abdbc6d1ff2acd8aac7a333e74f7c1da96048c36092765a7"}
Sep 11 17:06:09.020240 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:09.019880    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:06:09.020240 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:09.019903    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" event={"ID":"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3","Type":"ContainerStarted","Data":"df784d271e0888342d38338db55b29a6ba80331985ed4f5807c34d8ed318b3c1"}
Sep 11 17:06:09.034731 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:09.034679    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" podStartSLOduration=2.034660993 podStartE2EDuration="2.034660993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:06:09.033920237 +0000 UTC m=+2943.989967865" watchObservedRunningTime="2026-09-11 17:06:09.034660993 +0000 UTC m=+2943.990708623"
Sep 11 17:06:09.688925 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:09.688889    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:19.688235 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:19.688189    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:19.688738 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:19.688291    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:06:29.688658 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:29.688573    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:06:36.149749 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.149716    2550 generic.go:356] "Generic (PLEG): container finished" podID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerID="8d42c9dd6792892110882cdf2599efac46de6abdda7879c715aef0fb01878920" exitCode=0
Sep 11 17:06:36.150143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.149786    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" event={"ID":"d83693c4-da71-4241-bb1b-1276c22d6c26","Type":"ContainerDied","Data":"8d42c9dd6792892110882cdf2599efac46de6abdda7879c715aef0fb01878920"}
Sep 11 17:06:36.166534 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.166508    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:06:36.197252 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.197187    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhvzn\" (UniqueName: \"kubernetes.io/projected/d83693c4-da71-4241-bb1b-1276c22d6c26-kube-api-access-fhvzn\") pod \"d83693c4-da71-4241-bb1b-1276c22d6c26\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") "
Sep 11 17:06:36.197380 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.197265    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83693c4-da71-4241-bb1b-1276c22d6c26-tls-certs\") pod \"d83693c4-da71-4241-bb1b-1276c22d6c26\" (UID: \"d83693c4-da71-4241-bb1b-1276c22d6c26\") "
Sep 11 17:06:36.199338 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.199306    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83693c4-da71-4241-bb1b-1276c22d6c26-kube-api-access-fhvzn" (OuterVolumeSpecName: "kube-api-access-fhvzn") pod "d83693c4-da71-4241-bb1b-1276c22d6c26" (UID: "d83693c4-da71-4241-bb1b-1276c22d6c26"). InnerVolumeSpecName "kube-api-access-fhvzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:36.199443 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.199376    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83693c4-da71-4241-bb1b-1276c22d6c26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d83693c4-da71-4241-bb1b-1276c22d6c26" (UID: "d83693c4-da71-4241-bb1b-1276c22d6c26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:06:36.298424 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.298394    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvzn\" (UniqueName: \"kubernetes.io/projected/d83693c4-da71-4241-bb1b-1276c22d6c26-kube-api-access-fhvzn\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:36.298424 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:36.298419    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83693c4-da71-4241-bb1b-1276c22d6c26-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:37.155685 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:37.155651    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"
Sep 11 17:06:37.155685 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:37.155667    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm" event={"ID":"d83693c4-da71-4241-bb1b-1276c22d6c26","Type":"ContainerDied","Data":"d0fbb68bbd4b5fd3d8f27ad357eb9fd5c9009f251f899d6c0d552715d55633fc"}
Sep 11 17:06:37.156237 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:37.155724    2550 scope.go:117] "RemoveContainer" containerID="8d42c9dd6792892110882cdf2599efac46de6abdda7879c715aef0fb01878920"
Sep 11 17:06:37.177323 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:37.177297    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"]
Sep 11 17:06:37.181508 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:37.181484    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schew88pm"]
Sep 11 17:06:37.477757 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:37.477682    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" path="/var/lib/kubelet/pods/d83693c4-da71-4241-bb1b-1276c22d6c26/volumes"
Sep 11 17:06:38.664617 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:38.664586    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"]
Sep 11 17:06:38.672972 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:06:38.672936    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:06:38.672972 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:06:38.672972 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:06:38.672972 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885" cmd=["/bin/sleep","15"]
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:38.672982    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" message="" err=<
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:06:38.672995    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" podUID="963c03d6-7f17-41ed-a02a-f967481311ba" containerName="main" containerID="cri-o://7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"
Sep 11 17:06:38.673221 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:38.673032    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" podUID="963c03d6-7f17-41ed-a02a-f967481311ba" containerName="main" containerID="cri-o://7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885" gracePeriod=60
Sep 11 17:06:39.017512 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.017486    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:06:39.122386 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122359    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx6rs\" (UniqueName: \"kubernetes.io/projected/963c03d6-7f17-41ed-a02a-f967481311ba-kube-api-access-hx6rs\") pod \"963c03d6-7f17-41ed-a02a-f967481311ba\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") "
Sep 11 17:06:39.122571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122393    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-dshm\") pod \"963c03d6-7f17-41ed-a02a-f967481311ba\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") "
Sep 11 17:06:39.122571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122416    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-model-cache\") pod \"963c03d6-7f17-41ed-a02a-f967481311ba\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") "
Sep 11 17:06:39.122571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122458    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-tmp-dir\") pod \"963c03d6-7f17-41ed-a02a-f967481311ba\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") "
Sep 11 17:06:39.122571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122485    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-home\") pod \"963c03d6-7f17-41ed-a02a-f967481311ba\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") "
Sep 11 17:06:39.122571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122538    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963c03d6-7f17-41ed-a02a-f967481311ba-tls-certs\") pod \"963c03d6-7f17-41ed-a02a-f967481311ba\" (UID: \"963c03d6-7f17-41ed-a02a-f967481311ba\") "
Sep 11 17:06:39.122851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122771    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "963c03d6-7f17-41ed-a02a-f967481311ba" (UID: "963c03d6-7f17-41ed-a02a-f967481311ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:39.122851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122785    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-home" (OuterVolumeSpecName: "home") pod "963c03d6-7f17-41ed-a02a-f967481311ba" (UID: "963c03d6-7f17-41ed-a02a-f967481311ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:39.122851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.122802    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "963c03d6-7f17-41ed-a02a-f967481311ba" (UID: "963c03d6-7f17-41ed-a02a-f967481311ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:39.124524 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.124502    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-dshm" (OuterVolumeSpecName: "dshm") pod "963c03d6-7f17-41ed-a02a-f967481311ba" (UID: "963c03d6-7f17-41ed-a02a-f967481311ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:39.124620 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.124508    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963c03d6-7f17-41ed-a02a-f967481311ba-kube-api-access-hx6rs" (OuterVolumeSpecName: "kube-api-access-hx6rs") pod "963c03d6-7f17-41ed-a02a-f967481311ba" (UID: "963c03d6-7f17-41ed-a02a-f967481311ba"). InnerVolumeSpecName "kube-api-access-hx6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:39.124620 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.124523    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963c03d6-7f17-41ed-a02a-f967481311ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "963c03d6-7f17-41ed-a02a-f967481311ba" (UID: "963c03d6-7f17-41ed-a02a-f967481311ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:06:39.167792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.167762    2550 generic.go:356] "Generic (PLEG): container finished" podID="963c03d6-7f17-41ed-a02a-f967481311ba" containerID="7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885" exitCode=0
Sep 11 17:06:39.167908 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.167804    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"
Sep 11 17:06:39.167908 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.167831    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" event={"ID":"963c03d6-7f17-41ed-a02a-f967481311ba","Type":"ContainerDied","Data":"7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"}
Sep 11 17:06:39.167908 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.167882    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp" event={"ID":"963c03d6-7f17-41ed-a02a-f967481311ba","Type":"ContainerDied","Data":"9d4dffb5a5a0e1ab49af8476800bfd8f1a6e65c598ab2b853ce95274ec3c562c"}
Sep 11 17:06:39.168072 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.167914    2550 scope.go:117] "RemoveContainer" containerID="7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"
Sep 11 17:06:39.176747 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.176728    2550 scope.go:117] "RemoveContainer" containerID="7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"
Sep 11 17:06:39.177021 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:06:39.177002    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885\": container with ID starting with 7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885 not found: ID does not exist" containerID="7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"
Sep 11 17:06:39.177070 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.177026    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885"} err="failed to get container status \"7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885\": rpc error: code = NotFound desc = could not find container \"7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885\": container with ID starting with 7e8f021b4bfdd54a86de5cdc3fbf4acc4bc76b7a9394168d9f8a6b20a9897885 not found: ID does not exist"
Sep 11 17:06:39.187924 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.187870    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"]
Sep 11 17:06:39.193125 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.193106    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-gwgjp"]
Sep 11 17:06:39.223372 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.223355    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:39.223454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.223382    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:39.223454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.223391    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:39.223454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.223399    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963c03d6-7f17-41ed-a02a-f967481311ba-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:39.223454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.223408    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6rs\" (UniqueName: \"kubernetes.io/projected/963c03d6-7f17-41ed-a02a-f967481311ba-kube-api-access-hx6rs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:39.223454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.223417    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/963c03d6-7f17-41ed-a02a-f967481311ba-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:06:39.479792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:39.479709    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963c03d6-7f17-41ed-a02a-f967481311ba" path="/var/lib/kubelet/pods/963c03d6-7f17-41ed-a02a-f967481311ba/volumes"
Sep 11 17:06:40.026464 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:06:40.026432    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:07:05.676746 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:05.676719    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:07:05.686728 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:05.686706    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:07:43.935726 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.935683    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"]
Sep 11 17:07:43.936197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.936041    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963c03d6-7f17-41ed-a02a-f967481311ba" containerName="main"
Sep 11 17:07:43.936197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.936053    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="963c03d6-7f17-41ed-a02a-f967481311ba" containerName="main"
Sep 11 17:07:43.936197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.936076    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main"
Sep 11 17:07:43.936197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.936082    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main"
Sep 11 17:07:43.936197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.936137    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="963c03d6-7f17-41ed-a02a-f967481311ba" containerName="main"
Sep 11 17:07:43.936197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.936148    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d83693c4-da71-4241-bb1b-1276c22d6c26" containerName="main"
Sep 11 17:07:43.939024 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.939007    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:43.941498 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.941459    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 11 17:07:43.941498 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.941479    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 11 17:07:43.941710 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.941509    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 11 17:07:43.941710 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.941537    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-sfkpl\""
Sep 11 17:07:43.941710 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.941589    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-fz682\""
Sep 11 17:07:43.948589 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.948567    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"]
Sep 11 17:07:43.985853 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.985821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpnw\" (UniqueName: \"kubernetes.io/projected/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-kube-api-access-hgpnw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:43.985998 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:43.985920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:44.086210 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:44.086177    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:44.086396 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:44.086223    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpnw\" (UniqueName: \"kubernetes.io/projected/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-kube-api-access-hgpnw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:44.088791 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:44.088764    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:44.093808 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:44.093781    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpnw\" (UniqueName: \"kubernetes.io/projected/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-kube-api-access-hgpnw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:44.250300 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:44.250237    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:44.521593 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:44.521568    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"]
Sep 11 17:07:44.523832 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:07:44.523805    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c53c12c_fd40_4624_b5a8_8d1f9452fe97.slice/crio-3d0964e143e0a9a6f716bb8d519fd6226cf496c7531605596441c22db619140d WatchSource:0}: Error finding container 3d0964e143e0a9a6f716bb8d519fd6226cf496c7531605596441c22db619140d: Status 404 returned error can't find the container with id 3d0964e143e0a9a6f716bb8d519fd6226cf496c7531605596441c22db619140d
Sep 11 17:07:45.486222 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:45.486184    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" event={"ID":"0c53c12c-fd40-4624-b5a8-8d1f9452fe97","Type":"ContainerStarted","Data":"8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c"}
Sep 11 17:07:45.486656 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:45.486238    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" event={"ID":"0c53c12c-fd40-4624-b5a8-8d1f9452fe97","Type":"ContainerStarted","Data":"3d0964e143e0a9a6f716bb8d519fd6226cf496c7531605596441c22db619140d"}
Sep 11 17:07:45.486656 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:45.486259    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:07:45.503210 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:07:45.503158    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" podStartSLOduration=2.503144614 podStartE2EDuration="2.503144614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:07:45.50143709 +0000 UTC m=+3040.457484722" watchObservedRunningTime="2026-09-11 17:07:45.503144614 +0000 UTC m=+3040.459192242"
Sep 11 17:08:12.616813 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:12.616777    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"]
Sep 11 17:08:16.493103 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:16.493069    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:08:20.026523 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:20.026480    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:08:24.373869 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.373834    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"]
Sep 11 17:08:24.377714 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.377690    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.379660 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.379636    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 11 17:08:24.386699 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.386676    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"]
Sep 11 17:08:24.477652 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.477621    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.477791 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.477660    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvl4\" (UniqueName: \"kubernetes.io/projected/d9b44b80-f7b6-46b2-9b22-01392368b0e0-kube-api-access-ksvl4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.477791 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.477692    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.477791 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.477732    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.477930 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.477844    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.477930 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.477875    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578383 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578347    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578383 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578383    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578610 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578469    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578610 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578590    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578727 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578656    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578727 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578693    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvl4\" (UniqueName: \"kubernetes.io/projected/d9b44b80-f7b6-46b2-9b22-01392368b0e0-kube-api-access-ksvl4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578843 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578822    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578899 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578843    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.578961 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.578928    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.580965 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.580927    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.581252 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.581230    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.592745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.592719    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvl4\" (UniqueName: \"kubernetes.io/projected/d9b44b80-f7b6-46b2-9b22-01392368b0e0-kube-api-access-ksvl4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.689811 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.689726    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:24.967407 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:24.967378    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"]
Sep 11 17:08:24.969780 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:08:24.969756    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b44b80_f7b6_46b2_9b22_01392368b0e0.slice/crio-a6157185249beef4337b02d28f193655ef9c40728e762b98e01e9a709a923de8 WatchSource:0}: Error finding container a6157185249beef4337b02d28f193655ef9c40728e762b98e01e9a709a923de8: Status 404 returned error can't find the container with id a6157185249beef4337b02d28f193655ef9c40728e762b98e01e9a709a923de8
Sep 11 17:08:25.683996 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:25.683962    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" event={"ID":"d9b44b80-f7b6-46b2-9b22-01392368b0e0","Type":"ContainerStarted","Data":"3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb"}
Sep 11 17:08:25.684489 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:25.684029    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" event={"ID":"d9b44b80-f7b6-46b2-9b22-01392368b0e0","Type":"ContainerStarted","Data":"a6157185249beef4337b02d28f193655ef9c40728e762b98e01e9a709a923de8"}
Sep 11 17:08:25.689824 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:25.689791    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:25.717410 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:25.717362    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" podStartSLOduration=1.7173489910000002 podStartE2EDuration="1.717348991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:08:25.711834812 +0000 UTC m=+3080.667882440" watchObservedRunningTime="2026-09-11 17:08:25.717348991 +0000 UTC m=+3080.673396642"
Sep 11 17:08:27.635711 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:27.635667    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main" containerID="cri-o://1702871fec3605e1abdbc6d1ff2acd8aac7a333e74f7c1da96048c36092765a7" gracePeriod=45
Sep 11 17:08:30.026791 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:30.026750    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:08:34.690268 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:34.690228    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:34.703209 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:34.703186    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:34.737187 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:34.737162    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:40.026876 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:40.026833    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:08:40.027344 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:40.026944    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:08:50.026863 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:50.026822    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:08:57.842833 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:57.842799    2550 generic.go:356] "Generic (PLEG): container finished" podID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerID="1702871fec3605e1abdbc6d1ff2acd8aac7a333e74f7c1da96048c36092765a7" exitCode=0
Sep 11 17:08:57.843286 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:57.842880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" event={"ID":"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3","Type":"ContainerDied","Data":"1702871fec3605e1abdbc6d1ff2acd8aac7a333e74f7c1da96048c36092765a7"}
Sep 11 17:08:57.877892 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:57.877869    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:08:58.034374 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.034289    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r69qb\" (UniqueName: \"kubernetes.io/projected/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-kube-api-access-r69qb\") pod \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") "
Sep 11 17:08:58.034374 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.034338    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-tls-certs\") pod \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\" (UID: \"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3\") "
Sep 11 17:08:58.036573 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.036549    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" (UID: "c922fbdf-c5ce-49b2-93ad-c60e7031c5d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:08:58.036573 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.036535    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-kube-api-access-r69qb" (OuterVolumeSpecName: "kube-api-access-r69qb") pod "c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" (UID: "c922fbdf-c5ce-49b2-93ad-c60e7031c5d3"). InnerVolumeSpecName "kube-api-access-r69qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:08:58.135112 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.135079    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69qb\" (UniqueName: \"kubernetes.io/projected/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-kube-api-access-r69qb\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:58.135112 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.135108    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:58.542954 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.542919    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"]
Sep 11 17:08:58.552261 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:08:58.552222    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:08:58.552261 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:08:58.552261 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:08:58.552261 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb" cmd=["/bin/sleep","15"]
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.552281    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" message="" err=<
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:08:58.552299    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerName="main" containerID="cri-o://3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb"
Sep 11 17:08:58.552570 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.552397    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerName="main" containerID="cri-o://3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb" gracePeriod=60
Sep 11 17:08:58.727550 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.727509    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 11 17:08:58.850334 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.850302    2550 generic.go:356] "Generic (PLEG): container finished" podID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerID="3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb" exitCode=0
Sep 11 17:08:58.850720 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.850375    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" event={"ID":"d9b44b80-f7b6-46b2-9b22-01392368b0e0","Type":"ContainerDied","Data":"3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb"}
Sep 11 17:08:58.851772 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.851749    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl" event={"ID":"c922fbdf-c5ce-49b2-93ad-c60e7031c5d3","Type":"ContainerDied","Data":"df784d271e0888342d38338db55b29a6ba80331985ed4f5807c34d8ed318b3c1"}
Sep 11 17:08:58.851772 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.851765    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"
Sep 11 17:08:58.851922 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.851809    2550 scope.go:117] "RemoveContainer" containerID="1702871fec3605e1abdbc6d1ff2acd8aac7a333e74f7c1da96048c36092765a7"
Sep 11 17:08:58.874689 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.874665    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"]
Sep 11 17:08:58.877927 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.877908    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheflhsl"]
Sep 11 17:08:58.893919 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:08:58.893891    2550 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc922fbdf_c5ce_49b2_93ad_c60e7031c5d3.slice/crio-df784d271e0888342d38338db55b29a6ba80331985ed4f5807c34d8ed318b3c1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc922fbdf_c5ce_49b2_93ad_c60e7031c5d3.slice\": RecentStats: unable to find data in memory cache]"
Sep 11 17:08:58.898614 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:58.898595    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:59.042361 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042324    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksvl4\" (UniqueName: \"kubernetes.io/projected/d9b44b80-f7b6-46b2-9b22-01392368b0e0-kube-api-access-ksvl4\") pod \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") "
Sep 11 17:08:59.042361 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042361    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-model-cache\") pod \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") "
Sep 11 17:08:59.042626 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042400    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tmp-dir\") pod \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") "
Sep 11 17:08:59.042626 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042435    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tls-certs\") pod \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") "
Sep 11 17:08:59.042626 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042465    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-dshm\") pod \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") "
Sep 11 17:08:59.042626 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042482    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-home\") pod \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\" (UID: \"d9b44b80-f7b6-46b2-9b22-01392368b0e0\") "
Sep 11 17:08:59.042836 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042712    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9b44b80-f7b6-46b2-9b22-01392368b0e0" (UID: "d9b44b80-f7b6-46b2-9b22-01392368b0e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:59.042836 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042793    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9b44b80-f7b6-46b2-9b22-01392368b0e0" (UID: "d9b44b80-f7b6-46b2-9b22-01392368b0e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:59.042924 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.042838    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-home" (OuterVolumeSpecName: "home") pod "d9b44b80-f7b6-46b2-9b22-01392368b0e0" (UID: "d9b44b80-f7b6-46b2-9b22-01392368b0e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:59.044705 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.044675    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-dshm" (OuterVolumeSpecName: "dshm") pod "d9b44b80-f7b6-46b2-9b22-01392368b0e0" (UID: "d9b44b80-f7b6-46b2-9b22-01392368b0e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:59.044808 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.044731    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9b44b80-f7b6-46b2-9b22-01392368b0e0" (UID: "d9b44b80-f7b6-46b2-9b22-01392368b0e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:08:59.044808 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.044746    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b44b80-f7b6-46b2-9b22-01392368b0e0-kube-api-access-ksvl4" (OuterVolumeSpecName: "kube-api-access-ksvl4") pod "d9b44b80-f7b6-46b2-9b22-01392368b0e0" (UID: "d9b44b80-f7b6-46b2-9b22-01392368b0e0"). InnerVolumeSpecName "kube-api-access-ksvl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:08:59.143858 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.143826    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:59.143858 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.143853    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9b44b80-f7b6-46b2-9b22-01392368b0e0-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:59.144060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.143867    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:59.144060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.143879    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:59.144060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.143891    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvl4\" (UniqueName: \"kubernetes.io/projected/d9b44b80-f7b6-46b2-9b22-01392368b0e0-kube-api-access-ksvl4\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:59.144060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.143902    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9b44b80-f7b6-46b2-9b22-01392368b0e0-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:08:59.477048 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.476963    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" path="/var/lib/kubelet/pods/c922fbdf-c5ce-49b2-93ad-c60e7031c5d3/volumes"
Sep 11 17:08:59.864907 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.864863    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv" event={"ID":"d9b44b80-f7b6-46b2-9b22-01392368b0e0","Type":"ContainerDied","Data":"a6157185249beef4337b02d28f193655ef9c40728e762b98e01e9a709a923de8"}
Sep 11 17:08:59.865357 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.864914    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"
Sep 11 17:08:59.865357 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.864967    2550 scope.go:117] "RemoveContainer" containerID="3a47a2309f06e00bbee376192b9918ed5015fddee6c48f85f4dfebfe87d1eaeb"
Sep 11 17:08:59.903837 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.903807    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"]
Sep 11 17:08:59.905665 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:08:59.905645    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-xwplv"]
Sep 11 17:09:01.477857 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:01.477825    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" path="/var/lib/kubelet/pods/d9b44b80-f7b6-46b2-9b22-01392368b0e0/volumes"
Sep 11 17:09:05.079060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079023    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"]
Sep 11 17:09:05.079439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079360    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerName="main"
Sep 11 17:09:05.079439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079372    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerName="main"
Sep 11 17:09:05.079439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079380    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main"
Sep 11 17:09:05.079439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079385    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main"
Sep 11 17:09:05.079607 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079452    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c922fbdf-c5ce-49b2-93ad-c60e7031c5d3" containerName="main"
Sep 11 17:09:05.079607 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.079459    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9b44b80-f7b6-46b2-9b22-01392368b0e0" containerName="main"
Sep 11 17:09:05.084042 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.084002    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.086849 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.086825    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 11 17:09:05.087102 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.086830    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 11 17:09:05.087240 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.086871    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-tjf7g\""
Sep 11 17:09:05.087240 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.086874    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-b88bj\""
Sep 11 17:09:05.087240 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.086873    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 11 17:09:05.090855 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.090833    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"]
Sep 11 17:09:05.191248 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.191218    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnlp\" (UniqueName: \"kubernetes.io/projected/35457af2-cca1-43ab-b79e-7d4e6c67ea96-kube-api-access-slnlp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.191384 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.191268    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35457af2-cca1-43ab-b79e-7d4e6c67ea96-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.292080 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.292049    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnlp\" (UniqueName: \"kubernetes.io/projected/35457af2-cca1-43ab-b79e-7d4e6c67ea96-kube-api-access-slnlp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.292189 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.292098    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35457af2-cca1-43ab-b79e-7d4e6c67ea96-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.294496 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.294476    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35457af2-cca1-43ab-b79e-7d4e6c67ea96-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.299353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.299333    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnlp\" (UniqueName: \"kubernetes.io/projected/35457af2-cca1-43ab-b79e-7d4e6c67ea96-kube-api-access-slnlp\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.397493 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.397428    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-b88bj\""
Sep 11 17:09:05.407480 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.407459    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-tjf7g\""
Sep 11 17:09:05.415988 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.415972    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.685886 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.685862    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"]
Sep 11 17:09:05.688315 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:09:05.688287    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35457af2_cca1_43ab_b79e_7d4e6c67ea96.slice/crio-53d4dca005beec6b0b74249c619f9365cd6144a0e7d6938b2bafacbeecd42f87 WatchSource:0}: Error finding container 53d4dca005beec6b0b74249c619f9365cd6144a0e7d6938b2bafacbeecd42f87: Status 404 returned error can't find the container with id 53d4dca005beec6b0b74249c619f9365cd6144a0e7d6938b2bafacbeecd42f87
Sep 11 17:09:05.900650 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.900615    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" event={"ID":"35457af2-cca1-43ab-b79e-7d4e6c67ea96","Type":"ContainerStarted","Data":"2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815"}
Sep 11 17:09:05.900797 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.900688    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" event={"ID":"35457af2-cca1-43ab-b79e-7d4e6c67ea96","Type":"ContainerStarted","Data":"53d4dca005beec6b0b74249c619f9365cd6144a0e7d6938b2bafacbeecd42f87"}
Sep 11 17:09:05.900797 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.900742    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:05.917119 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:05.917073    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podStartSLOduration=0.917059509 podStartE2EDuration="917.059509ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:09:05.914260908 +0000 UTC m=+3120.870308536" watchObservedRunningTime="2026-09-11 17:09:05.917059509 +0000 UTC m=+3120.873107136"
Sep 11 17:09:36.908067 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:36.907985    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:09:48.734305 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:48.734271    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"]
Sep 11 17:09:56.492859 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:09:56.492819    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:10:03.752636 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:03.752592    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main" containerID="cri-o://8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c" gracePeriod=45
Sep 11 17:10:06.493014 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:06.492976    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:10:16.493110 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:16.493069    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:10:16.493537 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:16.493173    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:10:26.492396 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:26.492355    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:10:34.000634 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.000604    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:10:34.057150 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.057081    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgpnw\" (UniqueName: \"kubernetes.io/projected/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-kube-api-access-hgpnw\") pod \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") "
Sep 11 17:10:34.057150 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.057137    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-tls-certs\") pod \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\" (UID: \"0c53c12c-fd40-4624-b5a8-8d1f9452fe97\") "
Sep 11 17:10:34.059173 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.059142    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c53c12c-fd40-4624-b5a8-8d1f9452fe97" (UID: "0c53c12c-fd40-4624-b5a8-8d1f9452fe97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:10:34.059280 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.059142    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-kube-api-access-hgpnw" (OuterVolumeSpecName: "kube-api-access-hgpnw") pod "0c53c12c-fd40-4624-b5a8-8d1f9452fe97" (UID: "0c53c12c-fd40-4624-b5a8-8d1f9452fe97"). InnerVolumeSpecName "kube-api-access-hgpnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:10:34.158162 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.158134    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpnw\" (UniqueName: \"kubernetes.io/projected/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-kube-api-access-hgpnw\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:10:34.158162 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.158161    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c53c12c-fd40-4624-b5a8-8d1f9452fe97-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:10:34.322346 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.322279    2550 generic.go:356] "Generic (PLEG): container finished" podID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerID="8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c" exitCode=0
Sep 11 17:10:34.322346 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.322320    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"
Sep 11 17:10:34.322346 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.322330    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" event={"ID":"0c53c12c-fd40-4624-b5a8-8d1f9452fe97","Type":"ContainerDied","Data":"8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c"}
Sep 11 17:10:34.322562 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.322371    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq" event={"ID":"0c53c12c-fd40-4624-b5a8-8d1f9452fe97","Type":"ContainerDied","Data":"3d0964e143e0a9a6f716bb8d519fd6226cf496c7531605596441c22db619140d"}
Sep 11 17:10:34.322562 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.322392    2550 scope.go:117] "RemoveContainer" containerID="8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c"
Sep 11 17:10:34.331299 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.331275    2550 scope.go:117] "RemoveContainer" containerID="8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c"
Sep 11 17:10:34.331587 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:10:34.331563    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c\": container with ID starting with 8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c not found: ID does not exist" containerID="8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c"
Sep 11 17:10:34.331665 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.331594    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c"} err="failed to get container status \"8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c\": rpc error: code = NotFound desc = could not find container \"8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c\": container with ID starting with 8f4b78dbd8a8714464d5f4d3091235f86745cf25dddd4d30da92f148184bc06c not found: ID does not exist"
Sep 11 17:10:34.344697 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.344678    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"]
Sep 11 17:10:34.348223 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:34.348202    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schers2bq"]
Sep 11 17:10:35.476866 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:35.476829    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" path="/var/lib/kubelet/pods/0c53c12c-fd40-4624-b5a8-8d1f9452fe97/volumes"
Sep 11 17:10:53.282160 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.282121    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"]
Sep 11 17:10:53.282757 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.282738    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main"
Sep 11 17:10:53.282812 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.282761    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main"
Sep 11 17:10:53.282849 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.282836    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c53c12c-fd40-4624-b5a8-8d1f9452fe97" containerName="main"
Sep 11 17:10:53.285978 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.285957    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.288455 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.288428    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16251f0d-b883-44ed-ba67-99606bf92c02-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.288627 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.288493    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sm5j\" (UniqueName: \"kubernetes.io/projected/16251f0d-b883-44ed-ba67-99606bf92c02-kube-api-access-8sm5j\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.289237 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.289213    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 11 17:10:53.289353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.289252    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-s2hpk\""
Sep 11 17:10:53.289353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.289297    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-l44vr\""
Sep 11 17:10:53.289353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.289342    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 11 17:10:53.289563 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.289346    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 11 17:10:53.293011 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.292991    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"]
Sep 11 17:10:53.389310 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.389280    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sm5j\" (UniqueName: \"kubernetes.io/projected/16251f0d-b883-44ed-ba67-99606bf92c02-kube-api-access-8sm5j\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.389442 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.389321    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16251f0d-b883-44ed-ba67-99606bf92c02-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.391729 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.391703    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16251f0d-b883-44ed-ba67-99606bf92c02-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.396456 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.396436    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sm5j\" (UniqueName: \"kubernetes.io/projected/16251f0d-b883-44ed-ba67-99606bf92c02-kube-api-access-8sm5j\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.598866 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.598840    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:53.870088 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:53.870058    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"]
Sep 11 17:10:53.872028 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:10:53.872002    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16251f0d_b883_44ed_ba67_99606bf92c02.slice/crio-4732e2a24199a7083bd6a224cf23f9bba509c453782237729a8fcf8b39f3a719 WatchSource:0}: Error finding container 4732e2a24199a7083bd6a224cf23f9bba509c453782237729a8fcf8b39f3a719: Status 404 returned error can't find the container with id 4732e2a24199a7083bd6a224cf23f9bba509c453782237729a8fcf8b39f3a719
Sep 11 17:10:54.416532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:54.416496    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" event={"ID":"16251f0d-b883-44ed-ba67-99606bf92c02","Type":"ContainerStarted","Data":"5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677"}
Sep 11 17:10:54.416910 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:54.416562    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" event={"ID":"16251f0d-b883-44ed-ba67-99606bf92c02","Type":"ContainerStarted","Data":"4732e2a24199a7083bd6a224cf23f9bba509c453782237729a8fcf8b39f3a719"}
Sep 11 17:10:54.416910 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:54.416593    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:10:54.434298 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:10:54.434246    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podStartSLOduration=1.434232105 podStartE2EDuration="1.434232105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:10:54.432770771 +0000 UTC m=+3229.388818409" watchObservedRunningTime="2026-09-11 17:10:54.434232105 +0000 UTC m=+3229.390279732"
Sep 11 17:11:13.813586 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:13.813527    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"]
Sep 11 17:11:16.907494 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:16.907455    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:11:25.422948 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:25.422914    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:11:26.907386 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:26.907343    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:11:28.833775 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:28.833733    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main" containerID="cri-o://2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815" gracePeriod=45
Sep 11 17:11:36.907724 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:36.907681    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:11:36.908139 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:36.907820    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:11:46.907125 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:46.907088    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:11:56.907816 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:56.907776    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:11:59.086680 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.086653    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:11:59.163754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.163690    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slnlp\" (UniqueName: \"kubernetes.io/projected/35457af2-cca1-43ab-b79e-7d4e6c67ea96-kube-api-access-slnlp\") pod \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") "
Sep 11 17:11:59.163879 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.163775    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35457af2-cca1-43ab-b79e-7d4e6c67ea96-tls-certs\") pod \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\" (UID: \"35457af2-cca1-43ab-b79e-7d4e6c67ea96\") "
Sep 11 17:11:59.165801 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.165768    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35457af2-cca1-43ab-b79e-7d4e6c67ea96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35457af2-cca1-43ab-b79e-7d4e6c67ea96" (UID: "35457af2-cca1-43ab-b79e-7d4e6c67ea96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:11:59.165944 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.165824    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35457af2-cca1-43ab-b79e-7d4e6c67ea96-kube-api-access-slnlp" (OuterVolumeSpecName: "kube-api-access-slnlp") pod "35457af2-cca1-43ab-b79e-7d4e6c67ea96" (UID: "35457af2-cca1-43ab-b79e-7d4e6c67ea96"). InnerVolumeSpecName "kube-api-access-slnlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:11:59.264251 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.264219    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnlp\" (UniqueName: \"kubernetes.io/projected/35457af2-cca1-43ab-b79e-7d4e6c67ea96-kube-api-access-slnlp\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:11:59.264251 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.264248    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35457af2-cca1-43ab-b79e-7d4e6c67ea96-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:11:59.722183 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.722144    2550 generic.go:356] "Generic (PLEG): container finished" podID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerID="2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815" exitCode=0
Sep 11 17:11:59.722357 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.722198    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"
Sep 11 17:11:59.722357 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.722226    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" event={"ID":"35457af2-cca1-43ab-b79e-7d4e6c67ea96","Type":"ContainerDied","Data":"2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815"}
Sep 11 17:11:59.722357 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.722278    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn" event={"ID":"35457af2-cca1-43ab-b79e-7d4e6c67ea96","Type":"ContainerDied","Data":"53d4dca005beec6b0b74249c619f9365cd6144a0e7d6938b2bafacbeecd42f87"}
Sep 11 17:11:59.722357 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.722301    2550 scope.go:117] "RemoveContainer" containerID="2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815"
Sep 11 17:11:59.730681 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.730668    2550 scope.go:117] "RemoveContainer" containerID="2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815"
Sep 11 17:11:59.730984 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:11:59.730967    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815\": container with ID starting with 2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815 not found: ID does not exist" containerID="2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815"
Sep 11 17:11:59.731047 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.730991    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815"} err="failed to get container status \"2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815\": rpc error: code = NotFound desc = could not find container \"2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815\": container with ID starting with 2417a8c653af86bb6e7d7a6ef0e6415538e8b24a3edb9ce235418e6c4f465815 not found: ID does not exist"
Sep 11 17:11:59.738190 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.738142    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"]
Sep 11 17:11:59.744484 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:11:59.744464    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schegcgzn"]
Sep 11 17:12:01.477843 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:01.477810    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" path="/var/lib/kubelet/pods/35457af2-cca1-43ab-b79e-7d4e6c67ea96/volumes"
Sep 11 17:12:05.708528 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:05.708496    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:12:05.718379 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:05.718357    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:12:16.749940 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.749862    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"]
Sep 11 17:12:16.750462 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.750432    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main"
Sep 11 17:12:16.750462 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.750460    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main"
Sep 11 17:12:16.750675 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.750574    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="35457af2-cca1-43ab-b79e-7d4e6c67ea96" containerName="main"
Sep 11 17:12:16.755227 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.755206    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:16.757197 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.757175    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 11 17:12:16.757340 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.757316    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-28gqw\""
Sep 11 17:12:16.758690 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.758670    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 11 17:12:16.759137 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.759112    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 11 17:12:16.759668 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.759648    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-nt2s2\""
Sep 11 17:12:16.760797 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.760777    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"]
Sep 11 17:12:16.886234 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.886206    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxkk\" (UniqueName: \"kubernetes.io/projected/cec734b0-4239-41c8-81e0-d3b7988ace62-kube-api-access-2kxkk\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:16.886391 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.886249    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec734b0-4239-41c8-81e0-d3b7988ace62-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:16.986994 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.986964    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxkk\" (UniqueName: \"kubernetes.io/projected/cec734b0-4239-41c8-81e0-d3b7988ace62-kube-api-access-2kxkk\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:16.987128 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.987001    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec734b0-4239-41c8-81e0-d3b7988ace62-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:16.989457 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.989432    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec734b0-4239-41c8-81e0-d3b7988ace62-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:16.994830 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:16.994808    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxkk\" (UniqueName: \"kubernetes.io/projected/cec734b0-4239-41c8-81e0-d3b7988ace62-kube-api-access-2kxkk\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:17.068674 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:17.068621    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:17.340753 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:17.340726    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"]
Sep 11 17:12:17.342650 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:12:17.342620    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec734b0_4239_41c8_81e0_d3b7988ace62.slice/crio-b30ad14ff0444f788e16e3450afd4f07832c9db50dc042fc1ac80c74f7746376 WatchSource:0}: Error finding container b30ad14ff0444f788e16e3450afd4f07832c9db50dc042fc1ac80c74f7746376: Status 404 returned error can't find the container with id b30ad14ff0444f788e16e3450afd4f07832c9db50dc042fc1ac80c74f7746376
Sep 11 17:12:17.807856 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:17.807757    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" event={"ID":"cec734b0-4239-41c8-81e0-d3b7988ace62","Type":"ContainerStarted","Data":"f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982"}
Sep 11 17:12:17.807856 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:17.807816    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" event={"ID":"cec734b0-4239-41c8-81e0-d3b7988ace62","Type":"ContainerStarted","Data":"b30ad14ff0444f788e16e3450afd4f07832c9db50dc042fc1ac80c74f7746376"}
Sep 11 17:12:17.808291 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:17.807869    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:17.826650 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:17.826599    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" podStartSLOduration=1.82658502 podStartE2EDuration="1.82658502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:12:17.824933398 +0000 UTC m=+3312.780981027" watchObservedRunningTime="2026-09-11 17:12:17.82658502 +0000 UTC m=+3312.782632649"
Sep 11 17:12:44.316913 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:44.316877    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"]
Sep 11 17:12:45.423505 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:45.423465    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:12:48.813990 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:48.813910    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:12:55.422812 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:55.422774    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:12:59.337211 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:12:59.337155    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main" containerID="cri-o://5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677" gracePeriod=45
Sep 11 17:13:05.423494 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:05.423455    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:13:05.423970 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:05.423575    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:13:15.423180 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:15.423141    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:13:25.423412 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:25.423379    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:13:29.578297 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.578271    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:13:29.711034 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.710954    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16251f0d-b883-44ed-ba67-99606bf92c02-tls-certs\") pod \"16251f0d-b883-44ed-ba67-99606bf92c02\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") "
Sep 11 17:13:29.711034 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.710994    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sm5j\" (UniqueName: \"kubernetes.io/projected/16251f0d-b883-44ed-ba67-99606bf92c02-kube-api-access-8sm5j\") pod \"16251f0d-b883-44ed-ba67-99606bf92c02\" (UID: \"16251f0d-b883-44ed-ba67-99606bf92c02\") "
Sep 11 17:13:29.713191 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.713165    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16251f0d-b883-44ed-ba67-99606bf92c02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16251f0d-b883-44ed-ba67-99606bf92c02" (UID: "16251f0d-b883-44ed-ba67-99606bf92c02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:13:29.713280 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.713219    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16251f0d-b883-44ed-ba67-99606bf92c02-kube-api-access-8sm5j" (OuterVolumeSpecName: "kube-api-access-8sm5j") pod "16251f0d-b883-44ed-ba67-99606bf92c02" (UID: "16251f0d-b883-44ed-ba67-99606bf92c02"). InnerVolumeSpecName "kube-api-access-8sm5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:13:29.811851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.811826    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sm5j\" (UniqueName: \"kubernetes.io/projected/16251f0d-b883-44ed-ba67-99606bf92c02-kube-api-access-8sm5j\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:13:29.811851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:29.811851    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16251f0d-b883-44ed-ba67-99606bf92c02-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:13:30.153163 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.153128    2550 generic.go:356] "Generic (PLEG): container finished" podID="16251f0d-b883-44ed-ba67-99606bf92c02" containerID="5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677" exitCode=0
Sep 11 17:13:30.153296 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.153170    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"
Sep 11 17:13:30.153296 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.153211    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" event={"ID":"16251f0d-b883-44ed-ba67-99606bf92c02","Type":"ContainerDied","Data":"5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677"}
Sep 11 17:13:30.153296 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.153260    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p" event={"ID":"16251f0d-b883-44ed-ba67-99606bf92c02","Type":"ContainerDied","Data":"4732e2a24199a7083bd6a224cf23f9bba509c453782237729a8fcf8b39f3a719"}
Sep 11 17:13:30.153296 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.153285    2550 scope.go:117] "RemoveContainer" containerID="5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677"
Sep 11 17:13:30.162013 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.161982    2550 scope.go:117] "RemoveContainer" containerID="5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677"
Sep 11 17:13:30.162344 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:13:30.162322    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677\": container with ID starting with 5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677 not found: ID does not exist" containerID="5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677"
Sep 11 17:13:30.162404 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.162353    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677"} err="failed to get container status \"5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677\": rpc error: code = NotFound desc = could not find container \"5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677\": container with ID starting with 5da89b75e986ae7766d499948e4725afa921c6980d04fee07e32996694c73677 not found: ID does not exist"
Sep 11 17:13:30.173988 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.173965    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"]
Sep 11 17:13:30.178270 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:30.178250    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schewb77p"]
Sep 11 17:13:31.476475 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:31.476444    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" path="/var/lib/kubelet/pods/16251f0d-b883-44ed-ba67-99606bf92c02/volumes"
Sep 11 17:13:49.020235 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.020199    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"]
Sep 11 17:13:49.020743 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.020717    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main"
Sep 11 17:13:49.020743 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.020738    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main"
Sep 11 17:13:49.020884 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.020849    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="16251f0d-b883-44ed-ba67-99606bf92c02" containerName="main"
Sep 11 17:13:49.025374 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.025351    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.027952 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.027749    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 11 17:13:49.027952 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.027749    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 11 17:13:49.027952 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.027751    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-bmrxx\""
Sep 11 17:13:49.027952 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.027752    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-lv2cl\""
Sep 11 17:13:49.027952 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.027866    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 11 17:13:49.035132 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.035106    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"]
Sep 11 17:13:49.145617 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.145588    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5226e3a-53cf-47b3-9c05-1417f848e024-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.145769 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.145714    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p459\" (UniqueName: \"kubernetes.io/projected/e5226e3a-53cf-47b3-9c05-1417f848e024-kube-api-access-5p459\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.246837 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.246809    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5226e3a-53cf-47b3-9c05-1417f848e024-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.246987 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.246881    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p459\" (UniqueName: \"kubernetes.io/projected/e5226e3a-53cf-47b3-9c05-1417f848e024-kube-api-access-5p459\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.249277 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.249252    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5226e3a-53cf-47b3-9c05-1417f848e024-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.255305 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.255277    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p459\" (UniqueName: \"kubernetes.io/projected/e5226e3a-53cf-47b3-9c05-1417f848e024-kube-api-access-5p459\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.337351 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.337328    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:49.605813 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:49.605784    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"]
Sep 11 17:13:49.607459 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:13:49.607433    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5226e3a_53cf_47b3_9c05_1417f848e024.slice/crio-50d8e02aa0a5920dd4a39ca1929a292cd6e5b7826258ddd5fe32976e43cfbeb6 WatchSource:0}: Error finding container 50d8e02aa0a5920dd4a39ca1929a292cd6e5b7826258ddd5fe32976e43cfbeb6: Status 404 returned error can't find the container with id 50d8e02aa0a5920dd4a39ca1929a292cd6e5b7826258ddd5fe32976e43cfbeb6
Sep 11 17:13:50.247866 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:50.247832    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" event={"ID":"e5226e3a-53cf-47b3-9c05-1417f848e024","Type":"ContainerStarted","Data":"8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b"}
Sep 11 17:13:50.248276 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:50.247883    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" event={"ID":"e5226e3a-53cf-47b3-9c05-1417f848e024","Type":"ContainerStarted","Data":"50d8e02aa0a5920dd4a39ca1929a292cd6e5b7826258ddd5fe32976e43cfbeb6"}
Sep 11 17:13:50.248276 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:50.247940    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:13:50.267504 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:13:50.267454    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podStartSLOduration=1.26743942 podStartE2EDuration="1.26743942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:13:50.265672987 +0000 UTC m=+3405.221720615" watchObservedRunningTime="2026-09-11 17:13:50.26743942 +0000 UTC m=+3405.223487047"
Sep 11 17:14:21.255378 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:21.255347    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:14:22.242243 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:22.242210    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"]
Sep 11 17:14:28.814227 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:28.814188    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:14:37.266836 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:37.266792    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main" containerID="cri-o://f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982" gracePeriod=45
Sep 11 17:14:38.814446 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:38.814411    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:14:48.814187 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:48.814103    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:14:48.814564 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:48.814207    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:14:58.814107 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:14:58.814067    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:15:07.511963 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.511936    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:15:07.615976 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.615947    2550 generic.go:356] "Generic (PLEG): container finished" podID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerID="f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982" exitCode=0
Sep 11 17:15:07.616117 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.615988    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"
Sep 11 17:15:07.616117 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.615979    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" event={"ID":"cec734b0-4239-41c8-81e0-d3b7988ace62","Type":"ContainerDied","Data":"f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982"}
Sep 11 17:15:07.616117 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.616110    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff" event={"ID":"cec734b0-4239-41c8-81e0-d3b7988ace62","Type":"ContainerDied","Data":"b30ad14ff0444f788e16e3450afd4f07832c9db50dc042fc1ac80c74f7746376"}
Sep 11 17:15:07.616236 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.616135    2550 scope.go:117] "RemoveContainer" containerID="f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982"
Sep 11 17:15:07.624200 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.624185    2550 scope.go:117] "RemoveContainer" containerID="f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982"
Sep 11 17:15:07.624433 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:15:07.624415    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982\": container with ID starting with f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982 not found: ID does not exist" containerID="f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982"
Sep 11 17:15:07.624478 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.624443    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982"} err="failed to get container status \"f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982\": rpc error: code = NotFound desc = could not find container \"f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982\": container with ID starting with f2a73f697476ca02643666f1ebf9cf90af104a699f89777c209c2fadc45ca982 not found: ID does not exist"
Sep 11 17:15:07.630647 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.630630    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec734b0-4239-41c8-81e0-d3b7988ace62-tls-certs\") pod \"cec734b0-4239-41c8-81e0-d3b7988ace62\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") "
Sep 11 17:15:07.630704 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.630674    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kxkk\" (UniqueName: \"kubernetes.io/projected/cec734b0-4239-41c8-81e0-d3b7988ace62-kube-api-access-2kxkk\") pod \"cec734b0-4239-41c8-81e0-d3b7988ace62\" (UID: \"cec734b0-4239-41c8-81e0-d3b7988ace62\") "
Sep 11 17:15:07.632683 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.632664    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec734b0-4239-41c8-81e0-d3b7988ace62-kube-api-access-2kxkk" (OuterVolumeSpecName: "kube-api-access-2kxkk") pod "cec734b0-4239-41c8-81e0-d3b7988ace62" (UID: "cec734b0-4239-41c8-81e0-d3b7988ace62"). InnerVolumeSpecName "kube-api-access-2kxkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:15:07.632758 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.632729    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec734b0-4239-41c8-81e0-d3b7988ace62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cec734b0-4239-41c8-81e0-d3b7988ace62" (UID: "cec734b0-4239-41c8-81e0-d3b7988ace62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:15:07.731132 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.731098    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cec734b0-4239-41c8-81e0-d3b7988ace62-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:15:07.731132 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.731129    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxkk\" (UniqueName: \"kubernetes.io/projected/cec734b0-4239-41c8-81e0-d3b7988ace62-kube-api-access-2kxkk\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:15:07.936427 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.936394    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"]
Sep 11 17:15:07.942785 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:07.942761    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58jr8ff"]
Sep 11 17:15:09.477755 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:09.477715    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" path="/var/lib/kubelet/pods/cec734b0-4239-41c8-81e0-d3b7988ace62/volumes"
Sep 11 17:15:10.500494 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:10.500455    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"]
Sep 11 17:15:11.255620 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:11.255584    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:15:21.255831 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:21.255788    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:15:22.608239 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.608197    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"]
Sep 11 17:15:22.608713 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.608594    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main"
Sep 11 17:15:22.608713 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.608611    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main"
Sep 11 17:15:22.608713 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.608678    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec734b0-4239-41c8-81e0-d3b7988ace62" containerName="main"
Sep 11 17:15:22.611412 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.611395    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.613468 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.613445    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-qvc8f\""
Sep 11 17:15:22.614176 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.614158    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-v86fj\""
Sep 11 17:15:22.614462 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.614440    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 11 17:15:22.614576 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.614533    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 11 17:15:22.614659 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.614537    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 11 17:15:22.625023 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.625002    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"]
Sep 11 17:15:22.745186 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.745158    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfpz\" (UniqueName: \"kubernetes.io/projected/038c6d58-41fd-44b7-be5c-8bcf72a22603-kube-api-access-5lfpz\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.745360 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.745224    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038c6d58-41fd-44b7-be5c-8bcf72a22603-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.846604 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.846574    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038c6d58-41fd-44b7-be5c-8bcf72a22603-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.846781 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.846625    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfpz\" (UniqueName: \"kubernetes.io/projected/038c6d58-41fd-44b7-be5c-8bcf72a22603-kube-api-access-5lfpz\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.849104 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.849076    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038c6d58-41fd-44b7-be5c-8bcf72a22603-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.855040 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.855015    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfpz\" (UniqueName: \"kubernetes.io/projected/038c6d58-41fd-44b7-be5c-8bcf72a22603-kube-api-access-5lfpz\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:22.921905 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:22.921848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:23.203816 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:23.203732    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"]
Sep 11 17:15:23.207623 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:15:23.207592    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038c6d58_41fd_44b7_be5c_8bcf72a22603.slice/crio-fa8a5961421e0cdb1ef35b271aa4d94caccc9a6ba2f871c3a228f9995a87d35c WatchSource:0}: Error finding container fa8a5961421e0cdb1ef35b271aa4d94caccc9a6ba2f871c3a228f9995a87d35c: Status 404 returned error can't find the container with id fa8a5961421e0cdb1ef35b271aa4d94caccc9a6ba2f871c3a228f9995a87d35c
Sep 11 17:15:23.694718 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:23.694684    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" event={"ID":"038c6d58-41fd-44b7-be5c-8bcf72a22603","Type":"ContainerStarted","Data":"d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e"}
Sep 11 17:15:23.695162 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:23.694734    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" event={"ID":"038c6d58-41fd-44b7-be5c-8bcf72a22603","Type":"ContainerStarted","Data":"fa8a5961421e0cdb1ef35b271aa4d94caccc9a6ba2f871c3a228f9995a87d35c"}
Sep 11 17:15:23.695162 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:23.694804    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:23.709668 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:23.709493    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podStartSLOduration=1.7094781060000002 podStartE2EDuration="1.709478106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:23.708949137 +0000 UTC m=+3498.664996764" watchObservedRunningTime="2026-09-11 17:15:23.709478106 +0000 UTC m=+3498.665525737"
Sep 11 17:15:25.519517 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:25.519481    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main" containerID="cri-o://8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b" gracePeriod=45
Sep 11 17:15:31.255777 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:31.255695    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:15:31.256234 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:31.255805    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:15:41.255727 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:41.255674    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:15:51.255648 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:51.255593    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:15:54.701349 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:54.701317    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:15:55.766439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.766412    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:15:55.850645 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.850616    2550 generic.go:356] "Generic (PLEG): container finished" podID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerID="8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b" exitCode=0
Sep 11 17:15:55.850792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.850655    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"
Sep 11 17:15:55.850792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.850702    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" event={"ID":"e5226e3a-53cf-47b3-9c05-1417f848e024","Type":"ContainerDied","Data":"8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b"}
Sep 11 17:15:55.850792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.850750    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z" event={"ID":"e5226e3a-53cf-47b3-9c05-1417f848e024","Type":"ContainerDied","Data":"50d8e02aa0a5920dd4a39ca1929a292cd6e5b7826258ddd5fe32976e43cfbeb6"}
Sep 11 17:15:55.850792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.850774    2550 scope.go:117] "RemoveContainer" containerID="8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b"
Sep 11 17:15:55.859019 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.859001    2550 scope.go:117] "RemoveContainer" containerID="8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b"
Sep 11 17:15:55.859278 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:15:55.859262    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b\": container with ID starting with 8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b not found: ID does not exist" containerID="8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b"
Sep 11 17:15:55.859324 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.859285    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b"} err="failed to get container status \"8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b\": rpc error: code = NotFound desc = could not find container \"8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b\": container with ID starting with 8c1c905655da1a924a57c0e0ece328677d2ba9f47bf1dbf05d5dba86700c231b not found: ID does not exist"
Sep 11 17:15:55.900601 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.900582    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p459\" (UniqueName: \"kubernetes.io/projected/e5226e3a-53cf-47b3-9c05-1417f848e024-kube-api-access-5p459\") pod \"e5226e3a-53cf-47b3-9c05-1417f848e024\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") "
Sep 11 17:15:55.900704 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.900624    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5226e3a-53cf-47b3-9c05-1417f848e024-tls-certs\") pod \"e5226e3a-53cf-47b3-9c05-1417f848e024\" (UID: \"e5226e3a-53cf-47b3-9c05-1417f848e024\") "
Sep 11 17:15:55.902668 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.902639    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5226e3a-53cf-47b3-9c05-1417f848e024-kube-api-access-5p459" (OuterVolumeSpecName: "kube-api-access-5p459") pod "e5226e3a-53cf-47b3-9c05-1417f848e024" (UID: "e5226e3a-53cf-47b3-9c05-1417f848e024"). InnerVolumeSpecName "kube-api-access-5p459". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:15:55.902668 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:55.902658    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5226e3a-53cf-47b3-9c05-1417f848e024-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5226e3a-53cf-47b3-9c05-1417f848e024" (UID: "e5226e3a-53cf-47b3-9c05-1417f848e024"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:15:56.002060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:56.002030    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p459\" (UniqueName: \"kubernetes.io/projected/e5226e3a-53cf-47b3-9c05-1417f848e024-kube-api-access-5p459\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:15:56.002060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:56.002057    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5226e3a-53cf-47b3-9c05-1417f848e024-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:15:56.172350 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:56.172328    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"]
Sep 11 17:15:56.178432 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:56.178412    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheqrh5z"]
Sep 11 17:15:57.477077 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:15:57.477032    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" path="/var/lib/kubelet/pods/e5226e3a-53cf-47b3-9c05-1417f848e024/volumes"
Sep 11 17:16:26.462457 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.462414    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"]
Sep 11 17:16:26.462971 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.462937    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main"
Sep 11 17:16:26.462971 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.462959    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main"
Sep 11 17:16:26.463122 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.463059    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5226e3a-53cf-47b3-9c05-1417f848e024" containerName="main"
Sep 11 17:16:26.466227 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.466205    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.468191 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.468164    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 11 17:16:26.468491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.468468    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-jzs9d\""
Sep 11 17:16:26.468913 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.468896    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 11 17:16:26.468913 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.468901    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-n4wsg\""
Sep 11 17:16:26.469102 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.468945    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 11 17:16:26.476758 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.476735    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"]
Sep 11 17:16:26.513506 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.513480    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee87cdf-09bd-4e30-84c4-b42414b359ae-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.513646 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.513527    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqb7r\" (UniqueName: \"kubernetes.io/projected/0ee87cdf-09bd-4e30-84c4-b42414b359ae-kube-api-access-fqb7r\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.614882 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.614851    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee87cdf-09bd-4e30-84c4-b42414b359ae-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.615023 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.614904    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqb7r\" (UniqueName: \"kubernetes.io/projected/0ee87cdf-09bd-4e30-84c4-b42414b359ae-kube-api-access-fqb7r\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.617520 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.617497    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee87cdf-09bd-4e30-84c4-b42414b359ae-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.623183 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.623157    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqb7r\" (UniqueName: \"kubernetes.io/projected/0ee87cdf-09bd-4e30-84c4-b42414b359ae-kube-api-access-fqb7r\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:26.778469 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:26.778409    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:27.048246 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:27.048220    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"]
Sep 11 17:16:27.049559 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:16:27.049510    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee87cdf_09bd_4e30_84c4_b42414b359ae.slice/crio-157b46f6fcdc068e0b39d9dbd9343ae478fb0a658294a170859b2475978845a3 WatchSource:0}: Error finding container 157b46f6fcdc068e0b39d9dbd9343ae478fb0a658294a170859b2475978845a3: Status 404 returned error can't find the container with id 157b46f6fcdc068e0b39d9dbd9343ae478fb0a658294a170859b2475978845a3
Sep 11 17:16:28.005853 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:28.005814    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" event={"ID":"0ee87cdf-09bd-4e30-84c4-b42414b359ae","Type":"ContainerStarted","Data":"f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517"}
Sep 11 17:16:28.006293 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:28.005867    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" event={"ID":"0ee87cdf-09bd-4e30-84c4-b42414b359ae","Type":"ContainerStarted","Data":"157b46f6fcdc068e0b39d9dbd9343ae478fb0a658294a170859b2475978845a3"}
Sep 11 17:16:28.006293 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:28.005929    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:16:28.021653 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:28.021603    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" podStartSLOduration=2.02159028 podStartE2EDuration="2.02159028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:16:28.019405779 +0000 UTC m=+3562.975453406" watchObservedRunningTime="2026-09-11 17:16:28.02159028 +0000 UTC m=+3562.977637909"
Sep 11 17:16:30.205860 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:30.205824    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"]
Sep 11 17:16:45.225361 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:16:45.225306    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerName="main" containerID="cri-o://f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517" gracePeriod=45
Sep 11 17:17:00.209635 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:00.209536    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:17:05.740727 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:05.740699    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:17:05.752327 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:05.752301    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:17:10.207889 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:10.207852    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:17:15.474460 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.474437    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:17:15.575103 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.575018    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqb7r\" (UniqueName: \"kubernetes.io/projected/0ee87cdf-09bd-4e30-84c4-b42414b359ae-kube-api-access-fqb7r\") pod \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") "
Sep 11 17:17:15.575103 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.575084    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee87cdf-09bd-4e30-84c4-b42414b359ae-tls-certs\") pod \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\" (UID: \"0ee87cdf-09bd-4e30-84c4-b42414b359ae\") "
Sep 11 17:17:15.577264 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.577230    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee87cdf-09bd-4e30-84c4-b42414b359ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ee87cdf-09bd-4e30-84c4-b42414b359ae" (UID: "0ee87cdf-09bd-4e30-84c4-b42414b359ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:17:15.577408 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.577297    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee87cdf-09bd-4e30-84c4-b42414b359ae-kube-api-access-fqb7r" (OuterVolumeSpecName: "kube-api-access-fqb7r") pod "0ee87cdf-09bd-4e30-84c4-b42414b359ae" (UID: "0ee87cdf-09bd-4e30-84c4-b42414b359ae"). InnerVolumeSpecName "kube-api-access-fqb7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:17:15.676112 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.676081    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqb7r\" (UniqueName: \"kubernetes.io/projected/0ee87cdf-09bd-4e30-84c4-b42414b359ae-kube-api-access-fqb7r\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:17:15.676112 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:15.676107    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee87cdf-09bd-4e30-84c4-b42414b359ae-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:17:16.247423 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.247387    2550 generic.go:356] "Generic (PLEG): container finished" podID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerID="f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517" exitCode=0
Sep 11 17:17:16.247738 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.247441    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"
Sep 11 17:17:16.247738 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.247457    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" event={"ID":"0ee87cdf-09bd-4e30-84c4-b42414b359ae","Type":"ContainerDied","Data":"f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517"}
Sep 11 17:17:16.247738 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.247517    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4" event={"ID":"0ee87cdf-09bd-4e30-84c4-b42414b359ae","Type":"ContainerDied","Data":"157b46f6fcdc068e0b39d9dbd9343ae478fb0a658294a170859b2475978845a3"}
Sep 11 17:17:16.247738 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.247552    2550 scope.go:117] "RemoveContainer" containerID="f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517"
Sep 11 17:17:16.256814 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.256797    2550 scope.go:117] "RemoveContainer" containerID="f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517"
Sep 11 17:17:16.257070 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:17:16.257051    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517\": container with ID starting with f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517 not found: ID does not exist" containerID="f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517"
Sep 11 17:17:16.257115 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.257078    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517"} err="failed to get container status \"f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517\": rpc error: code = NotFound desc = could not find container \"f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517\": container with ID starting with f0bcba3b9951f2b38c212d6176d4d82a1127862f9ec282ae070d248c2922f517 not found: ID does not exist"
Sep 11 17:17:16.268619 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.268598    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"]
Sep 11 17:17:16.274520 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:16.274499    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-gr5t4"]
Sep 11 17:17:17.476998 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:17.476964    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" path="/var/lib/kubelet/pods/0ee87cdf-09bd-4e30-84c4-b42414b359ae/volumes"
Sep 11 17:17:28.169775 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.169731    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"]
Sep 11 17:17:28.170234 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.170223    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerName="main"
Sep 11 17:17:28.170277 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.170239    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerName="main"
Sep 11 17:17:28.170565 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.170335    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee87cdf-09bd-4e30-84c4-b42414b359ae" containerName="main"
Sep 11 17:17:28.173613 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.173594    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.175386 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.175366    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 11 17:17:28.175927 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.175907    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 11 17:17:28.176039 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.175951    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 11 17:17:28.176039 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.176026    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6xqjj\""
Sep 11 17:17:28.176144 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.176027    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-gczr5\""
Sep 11 17:17:28.181671 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.181395    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"]
Sep 11 17:17:28.268910 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.268879    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-kube-api-access-bgqrz\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.269050 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.268934    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.370473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.370442    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-kube-api-access-bgqrz\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.370609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.370510    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.373098 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.373076    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.378605 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.378582    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-kube-api-access-bgqrz\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.485956 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.485887    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:28.757274 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:28.757251    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"]
Sep 11 17:17:28.759100 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:17:28.759069    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48d0ef5_8fec_491c_bf07_c2a39af0fe93.slice/crio-e4572d5d37c0c0087040564e7b9ba5ea67646bd92b80b28b33e551259aa2a86c WatchSource:0}: Error finding container e4572d5d37c0c0087040564e7b9ba5ea67646bd92b80b28b33e551259aa2a86c: Status 404 returned error can't find the container with id e4572d5d37c0c0087040564e7b9ba5ea67646bd92b80b28b33e551259aa2a86c
Sep 11 17:17:29.312633 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:29.312595    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" event={"ID":"e48d0ef5-8fec-491c-bf07-c2a39af0fe93","Type":"ContainerStarted","Data":"61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174"}
Sep 11 17:17:29.313026 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:29.312682    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" event={"ID":"e48d0ef5-8fec-491c-bf07-c2a39af0fe93","Type":"ContainerStarted","Data":"e4572d5d37c0c0087040564e7b9ba5ea67646bd92b80b28b33e551259aa2a86c"}
Sep 11 17:17:29.313026 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:29.312706    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:17:29.329104 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:29.329054    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" podStartSLOduration=1.3290376369999999 podStartE2EDuration="1.329037637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:17:29.327497079 +0000 UTC m=+3624.283544706" watchObservedRunningTime="2026-09-11 17:17:29.329037637 +0000 UTC m=+3624.285085265"
Sep 11 17:17:54.296325 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:54.296291    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"]
Sep 11 17:17:54.700848 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:17:54.700813    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:18:00.319498 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:00.319461    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:18:04.701653 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:04.701608    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:18:09.317428 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:09.317390    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main" containerID="cri-o://d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e" gracePeriod=45
Sep 11 17:18:14.700599 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:14.700559    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:18:14.701008 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:14.700666    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:18:24.701026 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:24.700983    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:18:34.700982 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:34.700902    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:18:39.574497 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.574443    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:18:39.642587 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.642539    2550 generic.go:356] "Generic (PLEG): container finished" podID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerID="d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e" exitCode=0
Sep 11 17:18:39.642745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.642594    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"
Sep 11 17:18:39.642745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.642636    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" event={"ID":"038c6d58-41fd-44b7-be5c-8bcf72a22603","Type":"ContainerDied","Data":"d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e"}
Sep 11 17:18:39.642745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.642695    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894" event={"ID":"038c6d58-41fd-44b7-be5c-8bcf72a22603","Type":"ContainerDied","Data":"fa8a5961421e0cdb1ef35b271aa4d94caccc9a6ba2f871c3a228f9995a87d35c"}
Sep 11 17:18:39.642745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.642727    2550 scope.go:117] "RemoveContainer" containerID="d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e"
Sep 11 17:18:39.650971 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.650956    2550 scope.go:117] "RemoveContainer" containerID="d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e"
Sep 11 17:18:39.651220 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:18:39.651201    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e\": container with ID starting with d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e not found: ID does not exist" containerID="d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e"
Sep 11 17:18:39.651291 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.651226    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e"} err="failed to get container status \"d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e\": rpc error: code = NotFound desc = could not find container \"d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e\": container with ID starting with d9bb57eaee08af82ce0e473e1b3c739b423014378fb0eb6430851245594f1c4e not found: ID does not exist"
Sep 11 17:18:39.691585 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.691558    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lfpz\" (UniqueName: \"kubernetes.io/projected/038c6d58-41fd-44b7-be5c-8bcf72a22603-kube-api-access-5lfpz\") pod \"038c6d58-41fd-44b7-be5c-8bcf72a22603\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") "
Sep 11 17:18:39.691675 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.691589    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038c6d58-41fd-44b7-be5c-8bcf72a22603-tls-certs\") pod \"038c6d58-41fd-44b7-be5c-8bcf72a22603\" (UID: \"038c6d58-41fd-44b7-be5c-8bcf72a22603\") "
Sep 11 17:18:39.693659 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.693622    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038c6d58-41fd-44b7-be5c-8bcf72a22603-kube-api-access-5lfpz" (OuterVolumeSpecName: "kube-api-access-5lfpz") pod "038c6d58-41fd-44b7-be5c-8bcf72a22603" (UID: "038c6d58-41fd-44b7-be5c-8bcf72a22603"). InnerVolumeSpecName "kube-api-access-5lfpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:18:39.693771 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.693683    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038c6d58-41fd-44b7-be5c-8bcf72a22603-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "038c6d58-41fd-44b7-be5c-8bcf72a22603" (UID: "038c6d58-41fd-44b7-be5c-8bcf72a22603"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:18:39.792846 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.792822    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lfpz\" (UniqueName: \"kubernetes.io/projected/038c6d58-41fd-44b7-be5c-8bcf72a22603-kube-api-access-5lfpz\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:18:39.792846 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.792845    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038c6d58-41fd-44b7-be5c-8bcf72a22603-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:18:39.964207 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.964177    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"]
Sep 11 17:18:39.970062 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:39.970031    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-gd894"]
Sep 11 17:18:41.477439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:41.477394    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" path="/var/lib/kubelet/pods/038c6d58-41fd-44b7-be5c-8bcf72a22603/volumes"
Sep 11 17:18:45.899228 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.899188    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"]
Sep 11 17:18:45.899638 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.899565    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main"
Sep 11 17:18:45.899638 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.899578    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main"
Sep 11 17:18:45.899638 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.899635    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="038c6d58-41fd-44b7-be5c-8bcf72a22603" containerName="main"
Sep 11 17:18:45.903873 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.903849    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:45.906764 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.906709    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 11 17:18:45.907006 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.906716    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 11 17:18:45.907142 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.906766    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4zpqq\""
Sep 11 17:18:45.907142 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.906773    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 11 17:18:45.907142 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.906798    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-78wk5\""
Sep 11 17:18:45.911863 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:45.911837    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"]
Sep 11 17:18:46.038828 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.038796    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9zf\" (UniqueName: \"kubernetes.io/projected/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-kube-api-access-xg9zf\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.038997 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.038840    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.139477 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.139451    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9zf\" (UniqueName: \"kubernetes.io/projected/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-kube-api-access-xg9zf\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.139616 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.139482    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.141918 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.141897    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.149958 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.149906    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9zf\" (UniqueName: \"kubernetes.io/projected/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-kube-api-access-xg9zf\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.214808 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.214782    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.487752 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.487729    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"]
Sep 11 17:18:46.490201 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:18:46.490167    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode084d5a8_4ff5_4b10_bacb_81d24fefc94c.slice/crio-46ed86526afae2b94bbc336793d18e91b964b76d1d512dd3017d52320875f7c6 WatchSource:0}: Error finding container 46ed86526afae2b94bbc336793d18e91b964b76d1d512dd3017d52320875f7c6: Status 404 returned error can't find the container with id 46ed86526afae2b94bbc336793d18e91b964b76d1d512dd3017d52320875f7c6
Sep 11 17:18:46.679010 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.678972    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" event={"ID":"e084d5a8-4ff5-4b10-bacb-81d24fefc94c","Type":"ContainerStarted","Data":"75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4"}
Sep 11 17:18:46.679167 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.679025    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" event={"ID":"e084d5a8-4ff5-4b10-bacb-81d24fefc94c","Type":"ContainerStarted","Data":"46ed86526afae2b94bbc336793d18e91b964b76d1d512dd3017d52320875f7c6"}
Sep 11 17:18:46.679167 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.679054    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:18:46.697262 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:18:46.697214    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podStartSLOduration=1.697195791 podStartE2EDuration="1.697195791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:18:46.696273847 +0000 UTC m=+3701.652321477" watchObservedRunningTime="2026-09-11 17:18:46.697195791 +0000 UTC m=+3701.653243419"
Sep 11 17:19:17.685023 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:17.684986    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:19:30.477072 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:30.477038    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"]
Sep 11 17:19:39.008790 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.008751    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"]
Sep 11 17:19:39.012537 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.012519    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.014328 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.014304    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-cw8lp\""
Sep 11 17:19:39.019510 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.019487    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"]
Sep 11 17:19:39.129235 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.129188    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qbc\" (UniqueName: \"kubernetes.io/projected/af8e8554-6051-44bb-8ed8-c22bb465f76d-kube-api-access-82qbc\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.129390 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.129270    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8e8554-6051-44bb-8ed8-c22bb465f76d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.229855 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.229820    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8e8554-6051-44bb-8ed8-c22bb465f76d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.230011 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.229889    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qbc\" (UniqueName: \"kubernetes.io/projected/af8e8554-6051-44bb-8ed8-c22bb465f76d-kube-api-access-82qbc\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.232345 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.232316    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8e8554-6051-44bb-8ed8-c22bb465f76d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.237472 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.237453    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qbc\" (UniqueName: \"kubernetes.io/projected/af8e8554-6051-44bb-8ed8-c22bb465f76d-kube-api-access-82qbc\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.324656 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.324597    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.592886 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.592857    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"]
Sep 11 17:19:39.935005 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.934919    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" event={"ID":"af8e8554-6051-44bb-8ed8-c22bb465f76d","Type":"ContainerStarted","Data":"4a7a51ece251c5b059674d8f8f3f8281cc4df4450c737aed1fd94fa266b5f2f2"}
Sep 11 17:19:39.935005 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.934972    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" event={"ID":"af8e8554-6051-44bb-8ed8-c22bb465f76d","Type":"ContainerStarted","Data":"df2b21c06d8e222333aa0cc4c331d847b6baa1516c5092835d5bda10e830fdbc"}
Sep 11 17:19:39.935199 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.935066    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:19:39.951633 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:39.951575    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podStartSLOduration=1.9515634099999999 podStartE2EDuration="1.95156341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:19:39.948487794 +0000 UTC m=+3754.904535422" watchObservedRunningTime="2026-09-11 17:19:39.95156341 +0000 UTC m=+3754.907611038"
Sep 11 17:19:45.496767 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:45.496731    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main" containerID="cri-o://61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174" gracePeriod=45
Sep 11 17:19:50.319082 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:19:50.319041    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:20:00.318906 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:00.318820    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:20:10.319534 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:10.319494    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:20:10.320060 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:10.319622    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:20:10.942532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:10.942498    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:20:15.746922 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.746897    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:20:15.897710 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.897677    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-tls-certs\") pod \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") "
Sep 11 17:20:15.897879 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.897777    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-kube-api-access-bgqrz\") pod \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\" (UID: \"e48d0ef5-8fec-491c-bf07-c2a39af0fe93\") "
Sep 11 17:20:15.900021 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.899985    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-kube-api-access-bgqrz" (OuterVolumeSpecName: "kube-api-access-bgqrz") pod "e48d0ef5-8fec-491c-bf07-c2a39af0fe93" (UID: "e48d0ef5-8fec-491c-bf07-c2a39af0fe93"). InnerVolumeSpecName "kube-api-access-bgqrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:20:15.900143 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.900059    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e48d0ef5-8fec-491c-bf07-c2a39af0fe93" (UID: "e48d0ef5-8fec-491c-bf07-c2a39af0fe93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:20:15.999070 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.999048    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqrz\" (UniqueName: \"kubernetes.io/projected/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-kube-api-access-bgqrz\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:20:15.999070 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:15.999070    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d0ef5-8fec-491c-bf07-c2a39af0fe93-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:20:16.107669 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.107634    2550 generic.go:356] "Generic (PLEG): container finished" podID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerID="61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174" exitCode=0
Sep 11 17:20:16.107818 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.107684    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"
Sep 11 17:20:16.107818 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.107723    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" event={"ID":"e48d0ef5-8fec-491c-bf07-c2a39af0fe93","Type":"ContainerDied","Data":"61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174"}
Sep 11 17:20:16.107818 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.107779    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn" event={"ID":"e48d0ef5-8fec-491c-bf07-c2a39af0fe93","Type":"ContainerDied","Data":"e4572d5d37c0c0087040564e7b9ba5ea67646bd92b80b28b33e551259aa2a86c"}
Sep 11 17:20:16.107818 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.107801    2550 scope.go:117] "RemoveContainer" containerID="61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174"
Sep 11 17:20:16.116351 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.116334    2550 scope.go:117] "RemoveContainer" containerID="61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174"
Sep 11 17:20:16.116663 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:20:16.116634    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174\": container with ID starting with 61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174 not found: ID does not exist" containerID="61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174"
Sep 11 17:20:16.116746 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.116671    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174"} err="failed to get container status \"61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174\": rpc error: code = NotFound desc = could not find container \"61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174\": container with ID starting with 61384b9d2a18c6b2bdcea7cd69b3ca99ca698588cd852d3db1cba03fff34d174 not found: ID does not exist"
Sep 11 17:20:16.128320 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.128298    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"]
Sep 11 17:20:16.131996 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:16.131976    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-pfvrn"]
Sep 11 17:20:17.477573 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:17.477522    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" path="/var/lib/kubelet/pods/e48d0ef5-8fec-491c-bf07-c2a39af0fe93/volumes"
Sep 11 17:20:56.437650 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:56.437614    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"]
Sep 11 17:20:57.685847 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:20:57.685808    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:07.685031 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:07.684991    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:11.459397 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:11.459356    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main" containerID="cri-o://75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4" gracePeriod=45
Sep 11 17:21:17.685333 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:17.685294    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:17.685835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:17.685404    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:21:27.685232 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:27.685190    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:37.685524 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:37.685489    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:40.528965 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:40.528924    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"]
Sep 11 17:21:40.942231 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:40.942197    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:41.697038 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.697010    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:21:41.816715 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.816648    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg9zf\" (UniqueName: \"kubernetes.io/projected/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-kube-api-access-xg9zf\") pod \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") "
Sep 11 17:21:41.816715 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.816699    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-tls-certs\") pod \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\" (UID: \"e084d5a8-4ff5-4b10-bacb-81d24fefc94c\") "
Sep 11 17:21:41.818818 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.818783    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e084d5a8-4ff5-4b10-bacb-81d24fefc94c" (UID: "e084d5a8-4ff5-4b10-bacb-81d24fefc94c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:41.818926 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.818792    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-kube-api-access-xg9zf" (OuterVolumeSpecName: "kube-api-access-xg9zf") pod "e084d5a8-4ff5-4b10-bacb-81d24fefc94c" (UID: "e084d5a8-4ff5-4b10-bacb-81d24fefc94c"). InnerVolumeSpecName "kube-api-access-xg9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:41.917594 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.917567    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9zf\" (UniqueName: \"kubernetes.io/projected/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-kube-api-access-xg9zf\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:21:41.917594 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:41.917591    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e084d5a8-4ff5-4b10-bacb-81d24fefc94c-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:21:42.524024 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.523986    2550 generic.go:356] "Generic (PLEG): container finished" podID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerID="75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4" exitCode=0
Sep 11 17:21:42.524196 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.524033    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"
Sep 11 17:21:42.524196 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.524071    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" event={"ID":"e084d5a8-4ff5-4b10-bacb-81d24fefc94c","Type":"ContainerDied","Data":"75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4"}
Sep 11 17:21:42.524196 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.524122    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875" event={"ID":"e084d5a8-4ff5-4b10-bacb-81d24fefc94c","Type":"ContainerDied","Data":"46ed86526afae2b94bbc336793d18e91b964b76d1d512dd3017d52320875f7c6"}
Sep 11 17:21:42.524196 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.524146    2550 scope.go:117] "RemoveContainer" containerID="75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4"
Sep 11 17:21:42.532853 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.532834    2550 scope.go:117] "RemoveContainer" containerID="75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4"
Sep 11 17:21:42.533126 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:21:42.533108    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4\": container with ID starting with 75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4 not found: ID does not exist" containerID="75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4"
Sep 11 17:21:42.533174 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.533134    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4"} err="failed to get container status \"75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4\": rpc error: code = NotFound desc = could not find container \"75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4\": container with ID starting with 75c7f9ac9fc642cee91abf6ed3ef6383ca650269877f05d80682988b52e15da4 not found: ID does not exist"
Sep 11 17:21:42.544233 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.544212    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"]
Sep 11 17:21:42.548882 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:42.548858    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche7s875"]
Sep 11 17:21:43.477106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:43.477073    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" path="/var/lib/kubelet/pods/e084d5a8-4ff5-4b10-bacb-81d24fefc94c/volumes"
Sep 11 17:21:50.942168 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:50.942127    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:21:55.549779 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:21:55.549719    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main" containerID="cri-o://4a7a51ece251c5b059674d8f8f3f8281cc4df4450c737aed1fd94fa266b5f2f2" gracePeriod=45
Sep 11 17:22:00.942425 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:00.942387    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:22:00.942885 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:00.942491    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:22:05.770876 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.770846    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:22:05.784393 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.784372    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:22:05.863407 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.863376    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"]
Sep 11 17:22:05.863901 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.863869    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main"
Sep 11 17:22:05.863966 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.863907    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main"
Sep 11 17:22:05.863966 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.863921    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main"
Sep 11 17:22:05.863966 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.863929    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main"
Sep 11 17:22:05.864063 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.864033    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e084d5a8-4ff5-4b10-bacb-81d24fefc94c" containerName="main"
Sep 11 17:22:05.864063 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.864045    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48d0ef5-8fec-491c-bf07-c2a39af0fe93" containerName="main"
Sep 11 17:22:05.867297 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.867278    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:05.869955 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.869935    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 11 17:22:05.870087 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.869935    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-mfndp\""
Sep 11 17:22:05.870087 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.869981    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 11 17:22:05.870087 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.869989    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 11 17:22:05.870087 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.870033    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pmvlv\""
Sep 11 17:22:05.875500 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.875442    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"]
Sep 11 17:22:05.885492 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.885470    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfnz\" (UniqueName: \"kubernetes.io/projected/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-kube-api-access-tpfnz\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:05.885617 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.885510    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:05.985929 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.985907    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfnz\" (UniqueName: \"kubernetes.io/projected/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-kube-api-access-tpfnz\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:05.985994 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.985950    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:05.988475 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.988457    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:05.993897 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:05.993876    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfnz\" (UniqueName: \"kubernetes.io/projected/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-kube-api-access-tpfnz\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:06.179268 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:06.179230    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:06.455176 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:06.455095    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"]
Sep 11 17:22:06.459276 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:22:06.459251    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bfe7a4_cde3_44ce_98a8_7b852cab0cc1.slice/crio-2774a62fcb9747410e5e419a8831446eda059f7f26d1e82ef341cccea6f17054 WatchSource:0}: Error finding container 2774a62fcb9747410e5e419a8831446eda059f7f26d1e82ef341cccea6f17054: Status 404 returned error can't find the container with id 2774a62fcb9747410e5e419a8831446eda059f7f26d1e82ef341cccea6f17054
Sep 11 17:22:06.633183 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:06.633151    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" event={"ID":"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1","Type":"ContainerStarted","Data":"93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef"}
Sep 11 17:22:06.633340 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:06.633203    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" event={"ID":"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1","Type":"ContainerStarted","Data":"2774a62fcb9747410e5e419a8831446eda059f7f26d1e82ef341cccea6f17054"}
Sep 11 17:22:06.633340 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:06.633238    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:22:06.654600 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:06.654516    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" podStartSLOduration=1.6544978700000001 podStartE2EDuration="1.65449787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:06.653782795 +0000 UTC m=+3901.609830423" watchObservedRunningTime="2026-09-11 17:22:06.65449787 +0000 UTC m=+3901.610545500"
Sep 11 17:22:10.942823 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:10.942784    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:22:20.942858 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:20.942816    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:22:25.725279 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.725248    2550 generic.go:356] "Generic (PLEG): container finished" podID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerID="4a7a51ece251c5b059674d8f8f3f8281cc4df4450c737aed1fd94fa266b5f2f2" exitCode=0
Sep 11 17:22:25.725631 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.725309    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" event={"ID":"af8e8554-6051-44bb-8ed8-c22bb465f76d","Type":"ContainerDied","Data":"4a7a51ece251c5b059674d8f8f3f8281cc4df4450c737aed1fd94fa266b5f2f2"}
Sep 11 17:22:25.802181 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.802156    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:22:25.835315 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.835287    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82qbc\" (UniqueName: \"kubernetes.io/projected/af8e8554-6051-44bb-8ed8-c22bb465f76d-kube-api-access-82qbc\") pod \"af8e8554-6051-44bb-8ed8-c22bb465f76d\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") "
Sep 11 17:22:25.835447 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.835344    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8e8554-6051-44bb-8ed8-c22bb465f76d-tls-certs\") pod \"af8e8554-6051-44bb-8ed8-c22bb465f76d\" (UID: \"af8e8554-6051-44bb-8ed8-c22bb465f76d\") "
Sep 11 17:22:25.837308 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.837279    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8e8554-6051-44bb-8ed8-c22bb465f76d-kube-api-access-82qbc" (OuterVolumeSpecName: "kube-api-access-82qbc") pod "af8e8554-6051-44bb-8ed8-c22bb465f76d" (UID: "af8e8554-6051-44bb-8ed8-c22bb465f76d"). InnerVolumeSpecName "kube-api-access-82qbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:22:25.837425 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.837291    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af8e8554-6051-44bb-8ed8-c22bb465f76d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af8e8554-6051-44bb-8ed8-c22bb465f76d" (UID: "af8e8554-6051-44bb-8ed8-c22bb465f76d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:22:25.935873 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.935843    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af8e8554-6051-44bb-8ed8-c22bb465f76d-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:22:25.935873 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:25.935870    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qbc\" (UniqueName: \"kubernetes.io/projected/af8e8554-6051-44bb-8ed8-c22bb465f76d-kube-api-access-82qbc\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:22:26.731511 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:26.731476    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"
Sep 11 17:22:26.731511 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:26.731501    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn" event={"ID":"af8e8554-6051-44bb-8ed8-c22bb465f76d","Type":"ContainerDied","Data":"df2b21c06d8e222333aa0cc4c331d847b6baa1516c5092835d5bda10e830fdbc"}
Sep 11 17:22:26.732108 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:26.731583    2550 scope.go:117] "RemoveContainer" containerID="4a7a51ece251c5b059674d8f8f3f8281cc4df4450c737aed1fd94fa266b5f2f2"
Sep 11 17:22:26.752453 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:26.752428    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"]
Sep 11 17:22:26.758742 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:26.758720    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-glhfn"]
Sep 11 17:22:27.477098 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:27.477065    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" path="/var/lib/kubelet/pods/af8e8554-6051-44bb-8ed8-c22bb465f76d/volumes"
Sep 11 17:22:37.638919 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:22:37.638884    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:24:18.148448 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:18.148410    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"]
Sep 11 17:24:27.639601 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:27.639559    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:24:33.168985 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:33.168945    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main" containerID="cri-o://93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef" gracePeriod=45
Sep 11 17:24:37.639356 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:37.639315    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:24:47.639588 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:47.639527    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:24:47.640053 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:47.639663    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:24:57.639670 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:24:57.639629    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:25:03.418898 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.418873    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:25:03.460378 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.460310    2550 generic.go:356] "Generic (PLEG): container finished" podID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerID="93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef" exitCode=0
Sep 11 17:25:03.460378 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.460350    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"
Sep 11 17:25:03.460559 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.460384    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" event={"ID":"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1","Type":"ContainerDied","Data":"93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef"}
Sep 11 17:25:03.460559 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.460434    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2" event={"ID":"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1","Type":"ContainerDied","Data":"2774a62fcb9747410e5e419a8831446eda059f7f26d1e82ef341cccea6f17054"}
Sep 11 17:25:03.460559 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.460456    2550 scope.go:117] "RemoveContainer" containerID="93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef"
Sep 11 17:25:03.468631 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.468613    2550 scope.go:117] "RemoveContainer" containerID="93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef"
Sep 11 17:25:03.468884 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:25:03.468866    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef\": container with ID starting with 93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef not found: ID does not exist" containerID="93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef"
Sep 11 17:25:03.468948 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.468896    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef"} err="failed to get container status \"93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef\": rpc error: code = NotFound desc = could not find container \"93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef\": container with ID starting with 93f4692888ee53004afbc88fadc3d17cc2042341a07b4538768ebfedf9888bef not found: ID does not exist"
Sep 11 17:25:03.551711 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.551686    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpfnz\" (UniqueName: \"kubernetes.io/projected/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-kube-api-access-tpfnz\") pod \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") "
Sep 11 17:25:03.551819 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.551771    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-tls-certs\") pod \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\" (UID: \"18bfe7a4-cde3-44ce-98a8-7b852cab0cc1\") "
Sep 11 17:25:03.553736 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.553716    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" (UID: "18bfe7a4-cde3-44ce-98a8-7b852cab0cc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:25:03.553829 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.553780    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-kube-api-access-tpfnz" (OuterVolumeSpecName: "kube-api-access-tpfnz") pod "18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" (UID: "18bfe7a4-cde3-44ce-98a8-7b852cab0cc1"). InnerVolumeSpecName "kube-api-access-tpfnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:25:03.652790 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.652766    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:25:03.652790 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.652788    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfnz\" (UniqueName: \"kubernetes.io/projected/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1-kube-api-access-tpfnz\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:25:03.785149 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.785125    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"]
Sep 11 17:25:03.789105 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:03.789083    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48m6lw2"]
Sep 11 17:25:05.476858 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:05.476826    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" path="/var/lib/kubelet/pods/18bfe7a4-cde3-44ce-98a8-7b852cab0cc1/volumes"
Sep 11 17:25:08.563763 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.563726    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"]
Sep 11 17:25:08.564201 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.564184    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main"
Sep 11 17:25:08.564307 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.564204    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main"
Sep 11 17:25:08.564307 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.564234    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main"
Sep 11 17:25:08.564307 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.564243    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main"
Sep 11 17:25:08.564425 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.564321    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bfe7a4-cde3-44ce-98a8-7b852cab0cc1" containerName="main"
Sep 11 17:25:08.564425 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.564334    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="af8e8554-6051-44bb-8ed8-c22bb465f76d" containerName="main"
Sep 11 17:25:08.568844 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.568821    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.570668 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.570647    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 11 17:25:08.570860 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.570848    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 11 17:25:08.570927 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.570894    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:25:08.581040 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.581014    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"]
Sep 11 17:25:08.686610 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686581    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686615    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686643    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686665    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686681    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686701    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686965 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686777    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686965 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686792    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdwk\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-kube-api-access-fbdwk\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.686965 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.686815    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787645 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787612    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787785 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787659    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787775    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787851 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787826    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787961 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787857    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787961 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787883    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.787961 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787916    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788126 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787970    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788126 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.787994    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdwk\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-kube-api-access-fbdwk\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788247 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.788226    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788358 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.788326    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788470 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.788376    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788470 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.788448    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.788781 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.788762    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.790025 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.790003    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.790180 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.790164    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.794935 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.794913    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.795578 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.795537    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdwk\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-kube-api-access-fbdwk\") pod \"router-gateway-2-openshift-default-6cb79f7856-r48nn\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:08.880503 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:08.880485    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:09.155539 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:09.155511    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"]
Sep 11 17:25:09.157707 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:25:09.157676    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b8199_ce1c_4281_b150_8116a4b5a01f.slice/crio-f20d983026412073e674011f9075f8896e443e1f23ea61cf224271685c693882 WatchSource:0}: Error finding container f20d983026412073e674011f9075f8896e443e1f23ea61cf224271685c693882: Status 404 returned error can't find the container with id f20d983026412073e674011f9075f8896e443e1f23ea61cf224271685c693882
Sep 11 17:25:09.159744 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:09.159721    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:25:09.489039 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:09.488961    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" event={"ID":"7c3b8199-ce1c-4281-b150-8116a4b5a01f","Type":"ContainerStarted","Data":"f20d983026412073e674011f9075f8896e443e1f23ea61cf224271685c693882"}
Sep 11 17:25:12.083904 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:12.083850    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:25:12.084233 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:12.083952    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:25:12.084233 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:12.083995    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 17:25:12.507590 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:12.507495    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" event={"ID":"7c3b8199-ce1c-4281-b150-8116a4b5a01f","Type":"ContainerStarted","Data":"43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d"}
Sep 11 17:25:12.527996 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:12.527945    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podStartSLOduration=1.6041548479999999 podStartE2EDuration="4.527930756s" totalImagesPullingTime="2.923775908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:12.52519854 +0000 UTC m=+4087.481246169" watchObservedRunningTime="2026-09-11 17:25:12.527930756 +0000 UTC m=+4087.483978383"
Sep 11 17:25:12.880932 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:12.880893    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:13.882129 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:13.882079    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 11 17:25:14.881394 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:14.881351    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 11 17:25:15.881207 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:15.881165    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 11 17:25:16.881046 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:16.880994    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 11 17:25:17.881062 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:17.881018    2550 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Sep 11 17:25:18.884191 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:18.884160    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:18.884624 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:18.884511    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:18.885109 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:18.885090    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:25:32.062682 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.062648    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"]
Sep 11 17:25:32.065727 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.065705    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.067899 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.067872    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 11 17:25:32.068076 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.067872    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-cqq7h\""
Sep 11 17:25:32.068076 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.067876    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4527t\""
Sep 11 17:25:32.073752 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.073731    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"]
Sep 11 17:25:32.076238 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.076208    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.076391 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.076360    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45486\" (UniqueName: \"kubernetes.io/projected/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-kube-api-access-45486\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.177276 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.177244    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45486\" (UniqueName: \"kubernetes.io/projected/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-kube-api-access-45486\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.177449 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.177293    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.179758 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.179729    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.184862 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.184839    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45486\" (UniqueName: \"kubernetes.io/projected/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-kube-api-access-45486\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.378724 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.378687    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:32.648206 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:32.648179    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"]
Sep 11 17:25:32.649639 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:25:32.649604    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ba5f2c_8bde_40ce_ae15_5e97ebf5cd04.slice/crio-516ae6536ea87cf513abf0d443602f7391c60e8730b0607d11a34ff7ef508813 WatchSource:0}: Error finding container 516ae6536ea87cf513abf0d443602f7391c60e8730b0607d11a34ff7ef508813: Status 404 returned error can't find the container with id 516ae6536ea87cf513abf0d443602f7391c60e8730b0607d11a34ff7ef508813
Sep 11 17:25:33.599344 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:33.599306    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" event={"ID":"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04","Type":"ContainerStarted","Data":"4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65"}
Sep 11 17:25:33.599767 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:33.599367    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" event={"ID":"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04","Type":"ContainerStarted","Data":"516ae6536ea87cf513abf0d443602f7391c60e8730b0607d11a34ff7ef508813"}
Sep 11 17:25:33.599767 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:33.599494    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:25:33.614907 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:25:33.614865    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" podStartSLOduration=1.614850116 podStartE2EDuration="1.614850116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:33.613247407 +0000 UTC m=+4108.569295028" watchObservedRunningTime="2026-09-11 17:25:33.614850116 +0000 UTC m=+4108.570897745"
Sep 11 17:26:04.606693 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:26:04.606611    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:27:05.802595 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:05.802563    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:27:05.817746 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:05.817722    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:27:15.694645 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.694604    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"]
Sep 11 17:27:15.697109 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.697088    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:15.699013 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.698992    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 11 17:27:15.699498 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.699467    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 11 17:27:15.699498 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.699484    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 11 17:27:15.699710 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.699499    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-r67d6\""
Sep 11 17:27:15.699710 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.699467    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-kjwpf\""
Sep 11 17:27:15.705593 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.705571    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"]
Sep 11 17:27:15.748454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.748433    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3468b6fe-79d8-430f-87a8-3306f6f1ad66-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:15.748598 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.748462    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ntb5\" (UniqueName: \"kubernetes.io/projected/3468b6fe-79d8-430f-87a8-3306f6f1ad66-kube-api-access-2ntb5\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:15.848970 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.848927    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3468b6fe-79d8-430f-87a8-3306f6f1ad66-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:15.849194 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.848977    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ntb5\" (UniqueName: \"kubernetes.io/projected/3468b6fe-79d8-430f-87a8-3306f6f1ad66-kube-api-access-2ntb5\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:15.851804 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.851779    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3468b6fe-79d8-430f-87a8-3306f6f1ad66-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:15.856890 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:15.856869    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ntb5\" (UniqueName: \"kubernetes.io/projected/3468b6fe-79d8-430f-87a8-3306f6f1ad66-kube-api-access-2ntb5\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:16.009914 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:16.009834    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:16.283850 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:16.283823    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"]
Sep 11 17:27:16.285830 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:27:16.285804    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3468b6fe_79d8_430f_87a8_3306f6f1ad66.slice/crio-420420ee156eae9249a2b6c4249d852695e26f264b909482ee4fe2e74fc40fb9 WatchSource:0}: Error finding container 420420ee156eae9249a2b6c4249d852695e26f264b909482ee4fe2e74fc40fb9: Status 404 returned error can't find the container with id 420420ee156eae9249a2b6c4249d852695e26f264b909482ee4fe2e74fc40fb9
Sep 11 17:27:17.091111 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:17.091076    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" event={"ID":"3468b6fe-79d8-430f-87a8-3306f6f1ad66","Type":"ContainerStarted","Data":"d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137"}
Sep 11 17:27:17.091492 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:17.091128    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" event={"ID":"3468b6fe-79d8-430f-87a8-3306f6f1ad66","Type":"ContainerStarted","Data":"420420ee156eae9249a2b6c4249d852695e26f264b909482ee4fe2e74fc40fb9"}
Sep 11 17:27:17.091492 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:17.091174    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:17.117335 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:17.117288    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" podStartSLOduration=2.117271338 podStartE2EDuration="2.117271338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:17.115616052 +0000 UTC m=+4212.071663677" watchObservedRunningTime="2026-09-11 17:27:17.117271338 +0000 UTC m=+4212.073318966"
Sep 11 17:27:45.556362 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:45.556327    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"]
Sep 11 17:27:48.097609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:48.097577    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:27:54.606142 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:27:54.606093    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:28:00.579775 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:00.579721    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main" containerID="cri-o://4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65" gracePeriod=45
Sep 11 17:28:04.606379 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:04.606336    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:28:14.606260 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:14.606220    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:28:14.606657 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:14.606327    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:28:24.606091 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:24.606052    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:28:30.833653 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:30.833621    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:28:30.911802 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:30.911733    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-tls-certs\") pod \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") "
Sep 11 17:28:30.911802 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:30.911794    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45486\" (UniqueName: \"kubernetes.io/projected/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-kube-api-access-45486\") pod \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\" (UID: \"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04\") "
Sep 11 17:28:30.913942 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:30.913917    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" (UID: "28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:28:30.914022 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:30.913981    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-kube-api-access-45486" (OuterVolumeSpecName: "kube-api-access-45486") pod "28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" (UID: "28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04"). InnerVolumeSpecName "kube-api-access-45486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:31.013115 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.013088    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:31.013115 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.013112    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45486\" (UniqueName: \"kubernetes.io/projected/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04-kube-api-access-45486\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:31.446152 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.446117    2550 generic.go:356] "Generic (PLEG): container finished" podID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerID="4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65" exitCode=0
Sep 11 17:28:31.446317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.446157    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"
Sep 11 17:28:31.446317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.446178    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" event={"ID":"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04","Type":"ContainerDied","Data":"4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65"}
Sep 11 17:28:31.446317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.446218    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj" event={"ID":"28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04","Type":"ContainerDied","Data":"516ae6536ea87cf513abf0d443602f7391c60e8730b0607d11a34ff7ef508813"}
Sep 11 17:28:31.446317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.446240    2550 scope.go:117] "RemoveContainer" containerID="4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65"
Sep 11 17:28:31.455352 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.455332    2550 scope.go:117] "RemoveContainer" containerID="4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65"
Sep 11 17:28:31.455640 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:28:31.455614    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65\": container with ID starting with 4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65 not found: ID does not exist" containerID="4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65"
Sep 11 17:28:31.455688 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.455653    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65"} err="failed to get container status \"4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65\": rpc error: code = NotFound desc = could not find container \"4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65\": container with ID starting with 4671a03ac55a59ff3cf10f5a0b6168a64b9741f68f1243e2ee0593241f24bd65 not found: ID does not exist"
Sep 11 17:28:31.467950 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.467928    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"]
Sep 11 17:28:31.474607 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.474579    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45klrqj"]
Sep 11 17:28:31.481894 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:31.481861    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" path="/var/lib/kubelet/pods/28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04/volumes"
Sep 11 17:28:40.668675 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:40.668628    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"]
Sep 11 17:28:40.669185 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:40.668851    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy" containerID="cri-o://43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d" gracePeriod=30
Sep 11 17:28:42.781532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.781487    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"]
Sep 11 17:28:42.782058 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.782035    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main"
Sep 11 17:28:42.782149 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.782062    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main"
Sep 11 17:28:42.782198 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.782160    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ba5f2c-8bde-40ce-ae15-5e97ebf5cd04" containerName="main"
Sep 11 17:28:42.785046 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.785027    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:42.786792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.786765    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 11 17:28:42.787366 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.787341    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 11 17:28:42.787491 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.787403    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-5msph\""
Sep 11 17:28:42.787584 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.787487    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 11 17:28:42.787584 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.787567    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-wcpwz\""
Sep 11 17:28:42.795099 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.795076    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"]
Sep 11 17:28:42.896972 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.896920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:42.897156 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.896980    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9jp\" (UniqueName: \"kubernetes.io/projected/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-kube-api-access-kd9jp\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:42.998473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.998443    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:42.998473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:42.998475    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9jp\" (UniqueName: \"kubernetes.io/projected/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-kube-api-access-kd9jp\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:43.001118 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.001086    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:43.005835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.005815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9jp\" (UniqueName: \"kubernetes.io/projected/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-kube-api-access-kd9jp\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:43.097036 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.097012    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:43.366419 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.366394    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"]
Sep 11 17:28:43.367953 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:28:43.367923    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5ed3f2_791d_428a_9e1a_7d370027a1d6.slice/crio-60e4da20ee5ed1aafc09afd61b56e1efece2491ac5bcd46e6d34f1adef068494 WatchSource:0}: Error finding container 60e4da20ee5ed1aafc09afd61b56e1efece2491ac5bcd46e6d34f1adef068494: Status 404 returned error can't find the container with id 60e4da20ee5ed1aafc09afd61b56e1efece2491ac5bcd46e6d34f1adef068494
Sep 11 17:28:43.502437 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.502401    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" event={"ID":"2f5ed3f2-791d-428a-9e1a-7d370027a1d6","Type":"ContainerStarted","Data":"0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741"}
Sep 11 17:28:43.502622 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.502452    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" event={"ID":"2f5ed3f2-791d-428a-9e1a-7d370027a1d6","Type":"ContainerStarted","Data":"60e4da20ee5ed1aafc09afd61b56e1efece2491ac5bcd46e6d34f1adef068494"}
Sep 11 17:28:43.502622 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.502480    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:28:43.517796 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:43.517743    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podStartSLOduration=1.517726548 podStartE2EDuration="1.517726548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:28:43.515625753 +0000 UTC m=+4298.471673382" watchObservedRunningTime="2026-09-11 17:28:43.517726548 +0000 UTC m=+4298.473774227"
Sep 11 17:28:45.913587 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:45.913558    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:28:46.022031 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022001    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-data\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022168 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022038    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-podinfo\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022168 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022066    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-socket\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022168 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022097    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-token\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022168 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022117    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-credential-socket\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022344 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022194    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbdwk\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-kube-api-access-fbdwk\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022344 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022224    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istiod-ca-cert\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022344 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022256    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-envoy\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022344 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022290    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-certs\") pod \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\" (UID: \"7c3b8199-ce1c-4281-b150-8116a4b5a01f\") "
Sep 11 17:28:46.022562 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022283    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:46.022562 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022375    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:46.022562 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022485    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:46.022751 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.022725    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:46.023041 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.023017    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:28:46.024353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.024328    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:46.024353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.024337    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-kube-api-access-fbdwk" (OuterVolumeSpecName: "kube-api-access-fbdwk") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "kube-api-access-fbdwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:46.024609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.024583    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:28:46.024701 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.024649    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c3b8199-ce1c-4281-b150-8116a4b5a01f" (UID: "7c3b8199-ce1c-4281-b150-8116a4b5a01f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:46.123400 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123371    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123400 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123398    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-data\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123400 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123407    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-podinfo\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123632 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123414    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-workload-socket\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123632 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123422    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-token\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123632 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123430    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-credential-socket\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123632 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123440    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbdwk\" (UniqueName: \"kubernetes.io/projected/7c3b8199-ce1c-4281-b150-8116a4b5a01f-kube-api-access-fbdwk\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123632 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123448    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istiod-ca-cert\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.123632 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.123456    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3b8199-ce1c-4281-b150-8116a4b5a01f-istio-envoy\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:28:46.518149 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.518109    2550 generic.go:356] "Generic (PLEG): container finished" podID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerID="43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d" exitCode=0
Sep 11 17:28:46.518322 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.518188    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" event={"ID":"7c3b8199-ce1c-4281-b150-8116a4b5a01f","Type":"ContainerDied","Data":"43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d"}
Sep 11 17:28:46.518322 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.518204    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"
Sep 11 17:28:46.518322 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.518235    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn" event={"ID":"7c3b8199-ce1c-4281-b150-8116a4b5a01f","Type":"ContainerDied","Data":"f20d983026412073e674011f9075f8896e443e1f23ea61cf224271685c693882"}
Sep 11 17:28:46.518322 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.518256    2550 scope.go:117] "RemoveContainer" containerID="43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d"
Sep 11 17:28:46.527659 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.527640    2550 scope.go:117] "RemoveContainer" containerID="43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d"
Sep 11 17:28:46.527922 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:28:46.527904    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d\": container with ID starting with 43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d not found: ID does not exist" containerID="43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d"
Sep 11 17:28:46.527964 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.527931    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d"} err="failed to get container status \"43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d\": rpc error: code = NotFound desc = could not find container \"43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d\": container with ID starting with 43b138dcc89d034ab3a5d57564492f4586d985a1c590b548b302fa2e05aa484d not found: ID does not exist"
Sep 11 17:28:46.539744 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.539723    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"]
Sep 11 17:28:46.544998 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:46.544976    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-r48nn"]
Sep 11 17:28:47.477411 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:28:47.477377    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" path="/var/lib/kubelet/pods/7c3b8199-ce1c-4281-b150-8116a4b5a01f/volumes"
Sep 11 17:29:14.509215 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:14.509187    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:29:20.309799 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:20.309761    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"]
Sep 11 17:29:28.098610 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:28.098557    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:29:35.329515 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:35.329470    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main" containerID="cri-o://d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137" gracePeriod=45
Sep 11 17:29:38.097774 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:38.097737    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:29:48.098641 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:48.098598    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:29:48.099043 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:48.098711    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:29:58.098269 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:29:58.098230    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:30:05.581334 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.581304    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:30:05.596992 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.596898    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3468b6fe-79d8-430f-87a8-3306f6f1ad66-tls-certs\") pod \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") "
Sep 11 17:30:05.596992 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.596961    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ntb5\" (UniqueName: \"kubernetes.io/projected/3468b6fe-79d8-430f-87a8-3306f6f1ad66-kube-api-access-2ntb5\") pod \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\" (UID: \"3468b6fe-79d8-430f-87a8-3306f6f1ad66\") "
Sep 11 17:30:05.599695 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.599661    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3468b6fe-79d8-430f-87a8-3306f6f1ad66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3468b6fe-79d8-430f-87a8-3306f6f1ad66" (UID: "3468b6fe-79d8-430f-87a8-3306f6f1ad66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:30:05.599795 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.599731    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468b6fe-79d8-430f-87a8-3306f6f1ad66-kube-api-access-2ntb5" (OuterVolumeSpecName: "kube-api-access-2ntb5") pod "3468b6fe-79d8-430f-87a8-3306f6f1ad66" (UID: "3468b6fe-79d8-430f-87a8-3306f6f1ad66"). InnerVolumeSpecName "kube-api-access-2ntb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:30:05.698036 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.698009    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3468b6fe-79d8-430f-87a8-3306f6f1ad66-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:30:05.698036 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.698033    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ntb5\" (UniqueName: \"kubernetes.io/projected/3468b6fe-79d8-430f-87a8-3306f6f1ad66-kube-api-access-2ntb5\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:30:05.908372 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.908300    2550 generic.go:356] "Generic (PLEG): container finished" podID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerID="d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137" exitCode=0
Sep 11 17:30:05.908372 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.908344    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"
Sep 11 17:30:05.908527 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.908394    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" event={"ID":"3468b6fe-79d8-430f-87a8-3306f6f1ad66","Type":"ContainerDied","Data":"d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137"}
Sep 11 17:30:05.908527 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.908443    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb" event={"ID":"3468b6fe-79d8-430f-87a8-3306f6f1ad66","Type":"ContainerDied","Data":"420420ee156eae9249a2b6c4249d852695e26f264b909482ee4fe2e74fc40fb9"}
Sep 11 17:30:05.908527 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.908471    2550 scope.go:117] "RemoveContainer" containerID="d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137"
Sep 11 17:30:05.917155 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.917139    2550 scope.go:117] "RemoveContainer" containerID="d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137"
Sep 11 17:30:05.917496 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:30:05.917478    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137\": container with ID starting with d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137 not found: ID does not exist" containerID="d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137"
Sep 11 17:30:05.917575 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.917504    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137"} err="failed to get container status \"d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137\": rpc error: code = NotFound desc = could not find container \"d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137\": container with ID starting with d5f320f359a91e9dfe2124742a9a1b7afb0ad1fdea5740e9cfeb2b6b39639137 not found: ID does not exist"
Sep 11 17:30:05.929896 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.929864    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"]
Sep 11 17:30:05.933249 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:05.933228    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-qk6mb"]
Sep 11 17:30:07.476972 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:07.476939    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" path="/var/lib/kubelet/pods/3468b6fe-79d8-430f-87a8-3306f6f1ad66/volumes"
Sep 11 17:30:21.913066 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913033    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"]
Sep 11 17:30:21.913473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913363    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy"
Sep 11 17:30:21.913473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913378    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy"
Sep 11 17:30:21.913473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913389    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main"
Sep 11 17:30:21.913473 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913397    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main"
Sep 11 17:30:21.913716 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913485    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3b8199-ce1c-4281-b150-8116a4b5a01f" containerName="istio-proxy"
Sep 11 17:30:21.913716 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.913502    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3468b6fe-79d8-430f-87a8-3306f6f1ad66" containerName="main"
Sep 11 17:30:21.918061 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.918038    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:21.920238 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.920210    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-mzq97\""
Sep 11 17:30:21.920407 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.920226    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 11 17:30:21.920523 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.920231    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 11 17:30:21.920523 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.920281    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 11 17:30:21.920729 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.920344    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-4m94s\""
Sep 11 17:30:21.924677 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:21.924657    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"]
Sep 11 17:30:22.021146 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.021114    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.021291 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.021160    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6f9v\" (UniqueName: \"kubernetes.io/projected/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-kube-api-access-m6f9v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.121747 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.121721    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.121889 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.121765    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6f9v\" (UniqueName: \"kubernetes.io/projected/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-kube-api-access-m6f9v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.124314 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.124289    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.131015 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.130995    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6f9v\" (UniqueName: \"kubernetes.io/projected/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-kube-api-access-m6f9v\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.230562 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.230492    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:22.500665 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.500636    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"]
Sep 11 17:30:22.502795 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:30:22.502759    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67e0873_ec41_4dbf_92c5_a1f34e56d01c.slice/crio-d1ff9ed743eaf8a2a5e0f3dc60c4258b60a0b5cbd7317f3c7fcce92f43fc1182 WatchSource:0}: Error finding container d1ff9ed743eaf8a2a5e0f3dc60c4258b60a0b5cbd7317f3c7fcce92f43fc1182: Status 404 returned error can't find the container with id d1ff9ed743eaf8a2a5e0f3dc60c4258b60a0b5cbd7317f3c7fcce92f43fc1182
Sep 11 17:30:22.989917 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.989880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" event={"ID":"e67e0873-ec41-4dbf-92c5-a1f34e56d01c","Type":"ContainerStarted","Data":"aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71"}
Sep 11 17:30:22.990321 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.989942    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" event={"ID":"e67e0873-ec41-4dbf-92c5-a1f34e56d01c","Type":"ContainerStarted","Data":"d1ff9ed743eaf8a2a5e0f3dc60c4258b60a0b5cbd7317f3c7fcce92f43fc1182"}
Sep 11 17:30:22.990321 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:22.989981    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:30:23.010316 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:23.010262    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podStartSLOduration=2.010245208 podStartE2EDuration="2.010245208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:30:23.009664754 +0000 UTC m=+4397.965712393" watchObservedRunningTime="2026-09-11 17:30:23.010245208 +0000 UTC m=+4397.966292838"
Sep 11 17:30:53.996125 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:30:53.996095    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:31:02.713353 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.710789    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"]
Sep 11 17:31:02.715715 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.715682    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.721148 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.721118    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"]
Sep 11 17:31:02.823835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.823804    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.824004 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.823851    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894bec60-c8d9-455e-88e0-a9dddc70fd23-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.824004 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.823931    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn7h4\" (UniqueName: \"kubernetes.io/projected/894bec60-c8d9-455e-88e0-a9dddc70fd23-kube-api-access-nn7h4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.824004 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.823969    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.824004 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.823991    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.824195 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.824028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924510 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924477    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924678 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924517    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924678 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924678 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924608    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924678 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924643    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894bec60-c8d9-455e-88e0-a9dddc70fd23-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924849 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924685    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn7h4\" (UniqueName: \"kubernetes.io/projected/894bec60-c8d9-455e-88e0-a9dddc70fd23-kube-api-access-nn7h4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.924968 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924945    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.925013 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.924973    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.925062 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.925011    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.926948 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.926922    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.927246 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.927226    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894bec60-c8d9-455e-88e0-a9dddc70fd23-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:02.931633 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:02.931616    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn7h4\" (UniqueName: \"kubernetes.io/projected/894bec60-c8d9-455e-88e0-a9dddc70fd23-kube-api-access-nn7h4\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-72n6p\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:03.028364 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:03.028303    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:03.298520 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:03.298468    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"]
Sep 11 17:31:03.300746 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:31:03.300721    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894bec60_c8d9_455e_88e0_a9dddc70fd23.slice/crio-55d5f553ae393f548f6d35658ea01d4269a6711c0bae76b7b472758930cf598c WatchSource:0}: Error finding container 55d5f553ae393f548f6d35658ea01d4269a6711c0bae76b7b472758930cf598c: Status 404 returned error can't find the container with id 55d5f553ae393f548f6d35658ea01d4269a6711c0bae76b7b472758930cf598c
Sep 11 17:31:04.189483 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:04.189452    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" event={"ID":"894bec60-c8d9-455e-88e0-a9dddc70fd23","Type":"ContainerStarted","Data":"4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"}
Sep 11 17:31:04.189899 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:04.189503    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" event={"ID":"894bec60-c8d9-455e-88e0-a9dddc70fd23","Type":"ContainerStarted","Data":"55d5f553ae393f548f6d35658ea01d4269a6711c0bae76b7b472758930cf598c"}
Sep 11 17:31:04.204915 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:04.204870    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" podStartSLOduration=2.204854864 podStartE2EDuration="2.204854864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:31:04.203020287 +0000 UTC m=+4439.159067914" watchObservedRunningTime="2026-09-11 17:31:04.204854864 +0000 UTC m=+4439.160902491"
Sep 11 17:31:05.029394 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:05.029356    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:13.029502 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:13.029466    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:13.041819 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:13.041791    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:13.243514 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:13.243486    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:21.028146 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:21.028107    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"]
Sep 11 17:31:21.788098 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:21.788062    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"]
Sep 11 17:31:21.796610 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:31:21.796572    2550 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:31:21.796610 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:31:21.796610 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:31:21.796610 ip-10-0-141-84 kubenswrapper[2550]:  > containerID="4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28" cmd=["/bin/sleep","15"]
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:21.796628    2550 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" message="" err=<
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]:  >
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:31:21.796641    2550 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]:         , stdout: , stderr: , exit code -1
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" podUID="894bec60-c8d9-455e-88e0-a9dddc70fd23" containerName="main" containerID="cri-o://4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"
Sep 11 17:31:21.796865 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:21.796674    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" podUID="894bec60-c8d9-455e-88e0-a9dddc70fd23" containerName="main" containerID="cri-o://4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28" gracePeriod=60
Sep 11 17:31:21.815183 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:21.815154    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"]
Sep 11 17:31:22.140273 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.140248    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:22.280144 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280113    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-home\") pod \"894bec60-c8d9-455e-88e0-a9dddc70fd23\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") "
Sep 11 17:31:22.280317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280165    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-tmp-dir\") pod \"894bec60-c8d9-455e-88e0-a9dddc70fd23\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") "
Sep 11 17:31:22.280317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280186    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894bec60-c8d9-455e-88e0-a9dddc70fd23-tls-certs\") pod \"894bec60-c8d9-455e-88e0-a9dddc70fd23\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") "
Sep 11 17:31:22.280317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280219    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn7h4\" (UniqueName: \"kubernetes.io/projected/894bec60-c8d9-455e-88e0-a9dddc70fd23-kube-api-access-nn7h4\") pod \"894bec60-c8d9-455e-88e0-a9dddc70fd23\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") "
Sep 11 17:31:22.280317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280255    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-dshm\") pod \"894bec60-c8d9-455e-88e0-a9dddc70fd23\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") "
Sep 11 17:31:22.280317 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280272    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-model-cache\") pod \"894bec60-c8d9-455e-88e0-a9dddc70fd23\" (UID: \"894bec60-c8d9-455e-88e0-a9dddc70fd23\") "
Sep 11 17:31:22.280609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280374    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-home" (OuterVolumeSpecName: "home") pod "894bec60-c8d9-455e-88e0-a9dddc70fd23" (UID: "894bec60-c8d9-455e-88e0-a9dddc70fd23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:22.280609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280488    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-home\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:31:22.280723 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280631    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-model-cache" (OuterVolumeSpecName: "model-cache") pod "894bec60-c8d9-455e-88e0-a9dddc70fd23" (UID: "894bec60-c8d9-455e-88e0-a9dddc70fd23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:22.280792 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.280731    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "894bec60-c8d9-455e-88e0-a9dddc70fd23" (UID: "894bec60-c8d9-455e-88e0-a9dddc70fd23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:22.281106 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.281078    2550 generic.go:356] "Generic (PLEG): container finished" podID="894bec60-c8d9-455e-88e0-a9dddc70fd23" containerID="4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28" exitCode=0
Sep 11 17:31:22.281215 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.281128    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"
Sep 11 17:31:22.281215 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.281164    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" event={"ID":"894bec60-c8d9-455e-88e0-a9dddc70fd23","Type":"ContainerDied","Data":"4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"}
Sep 11 17:31:22.281327 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.281229    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p" event={"ID":"894bec60-c8d9-455e-88e0-a9dddc70fd23","Type":"ContainerDied","Data":"55d5f553ae393f548f6d35658ea01d4269a6711c0bae76b7b472758930cf598c"}
Sep 11 17:31:22.281327 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.281264    2550 scope.go:117] "RemoveContainer" containerID="4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"
Sep 11 17:31:22.282767 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.282736    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894bec60-c8d9-455e-88e0-a9dddc70fd23-kube-api-access-nn7h4" (OuterVolumeSpecName: "kube-api-access-nn7h4") pod "894bec60-c8d9-455e-88e0-a9dddc70fd23" (UID: "894bec60-c8d9-455e-88e0-a9dddc70fd23"). InnerVolumeSpecName "kube-api-access-nn7h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:22.282767 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.282755    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894bec60-c8d9-455e-88e0-a9dddc70fd23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "894bec60-c8d9-455e-88e0-a9dddc70fd23" (UID: "894bec60-c8d9-455e-88e0-a9dddc70fd23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:31:22.282928 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.282755    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-dshm" (OuterVolumeSpecName: "dshm") pod "894bec60-c8d9-455e-88e0-a9dddc70fd23" (UID: "894bec60-c8d9-455e-88e0-a9dddc70fd23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:22.297932 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.297916    2550 scope.go:117] "RemoveContainer" containerID="4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"
Sep 11 17:31:22.298175 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:31:22.298159    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28\": container with ID starting with 4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28 not found: ID does not exist" containerID="4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"
Sep 11 17:31:22.298225 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.298182    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28"} err="failed to get container status \"4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28\": rpc error: code = NotFound desc = could not find container \"4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28\": container with ID starting with 4320e540e09bb82b0bd2ddb482a2fd7119cd95b3d71e5c732f374f07af862a28 not found: ID does not exist"
Sep 11 17:31:22.381855 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.381826    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-dshm\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:31:22.381855 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.381851    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-model-cache\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:31:22.382014 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.381880    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/894bec60-c8d9-455e-88e0-a9dddc70fd23-tmp-dir\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:31:22.382014 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.381888    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894bec60-c8d9-455e-88e0-a9dddc70fd23-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:31:22.382014 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.381898    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn7h4\" (UniqueName: \"kubernetes.io/projected/894bec60-c8d9-455e-88e0-a9dddc70fd23-kube-api-access-nn7h4\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:31:22.602336 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.602304    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"]
Sep 11 17:31:22.605817 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:22.605794    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-72n6p"]
Sep 11 17:31:23.477454 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:23.477416    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894bec60-c8d9-455e-88e0-a9dddc70fd23" path="/var/lib/kubelet/pods/894bec60-c8d9-455e-88e0-a9dddc70fd23/volumes"
Sep 11 17:31:23.997188 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:23.997151    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:24.508609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:24.508571    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:33.996914 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:33.996876    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:34.508901 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:34.508863    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:36.047066 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:36.047016    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main" containerID="cri-o://0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741" gracePeriod=45
Sep 11 17:31:36.833285 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:36.833243    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main" containerID="cri-o://aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71" gracePeriod=45
Sep 11 17:31:43.996864 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:43.996828    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:43.997271 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:43.996942    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:31:44.508307 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:44.508271    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:44.508487 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:44.508371    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:31:53.997253 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:53.997213    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:31:54.508356 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:31:54.508321    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:32:03.996378 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:03.996290    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:32:04.508287 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:04.508254    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 17:32:05.837433 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:05.837400    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:32:05.848938 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:05.848918    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s6jwt_92ffbbfc-30e6-4dfe-815d-713685253804/ovn-acl-logging/0.log"
Sep 11 17:32:06.297201 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.297172    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:32:06.389267 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.389237    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-tls-certs\") pod \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") "
Sep 11 17:32:06.389430 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.389300    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd9jp\" (UniqueName: \"kubernetes.io/projected/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-kube-api-access-kd9jp\") pod \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\" (UID: \"2f5ed3f2-791d-428a-9e1a-7d370027a1d6\") "
Sep 11 17:32:06.391532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.391498    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-kube-api-access-kd9jp" (OuterVolumeSpecName: "kube-api-access-kd9jp") pod "2f5ed3f2-791d-428a-9e1a-7d370027a1d6" (UID: "2f5ed3f2-791d-428a-9e1a-7d370027a1d6"). InnerVolumeSpecName "kube-api-access-kd9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:06.391532 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.391526    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f5ed3f2-791d-428a-9e1a-7d370027a1d6" (UID: "2f5ed3f2-791d-428a-9e1a-7d370027a1d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:32:06.490261 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.490186    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:32:06.490261 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.490220    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9jp\" (UniqueName: \"kubernetes.io/projected/2f5ed3f2-791d-428a-9e1a-7d370027a1d6-kube-api-access-kd9jp\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:32:06.492401 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.492375    2550 generic.go:356] "Generic (PLEG): container finished" podID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerID="0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741" exitCode=0
Sep 11 17:32:06.492521 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.492419    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"
Sep 11 17:32:06.492521 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.492470    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" event={"ID":"2f5ed3f2-791d-428a-9e1a-7d370027a1d6","Type":"ContainerDied","Data":"0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741"}
Sep 11 17:32:06.492671 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.492529    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq" event={"ID":"2f5ed3f2-791d-428a-9e1a-7d370027a1d6","Type":"ContainerDied","Data":"60e4da20ee5ed1aafc09afd61b56e1efece2491ac5bcd46e6d34f1adef068494"}
Sep 11 17:32:06.492671 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.492579    2550 scope.go:117] "RemoveContainer" containerID="0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741"
Sep 11 17:32:06.501998 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.501971    2550 scope.go:117] "RemoveContainer" containerID="0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741"
Sep 11 17:32:06.502271 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:32:06.502253    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741\": container with ID starting with 0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741 not found: ID does not exist" containerID="0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741"
Sep 11 17:32:06.502315 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.502281    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741"} err="failed to get container status \"0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741\": rpc error: code = NotFound desc = could not find container \"0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741\": container with ID starting with 0836a8228172a25eb1718a370c9c4a0ba4914d1b67460bad1ea7f37997da6741 not found: ID does not exist"
Sep 11 17:32:06.513260 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.513235    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"]
Sep 11 17:32:06.519111 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:06.519090    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tlnsq"]
Sep 11 17:32:07.079520 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.079496    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:32:07.196567 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.196475    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6f9v\" (UniqueName: \"kubernetes.io/projected/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-kube-api-access-m6f9v\") pod \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") "
Sep 11 17:32:07.196702 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.196565    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-tls-certs\") pod \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\" (UID: \"e67e0873-ec41-4dbf-92c5-a1f34e56d01c\") "
Sep 11 17:32:07.198578 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.198549    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-kube-api-access-m6f9v" (OuterVolumeSpecName: "kube-api-access-m6f9v") pod "e67e0873-ec41-4dbf-92c5-a1f34e56d01c" (UID: "e67e0873-ec41-4dbf-92c5-a1f34e56d01c"). InnerVolumeSpecName "kube-api-access-m6f9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:32:07.198695 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.198634    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e67e0873-ec41-4dbf-92c5-a1f34e56d01c" (UID: "e67e0873-ec41-4dbf-92c5-a1f34e56d01c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:32:07.297666 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.297629    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6f9v\" (UniqueName: \"kubernetes.io/projected/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-kube-api-access-m6f9v\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:32:07.297666 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.297664    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67e0873-ec41-4dbf-92c5-a1f34e56d01c-tls-certs\") on node \"ip-10-0-141-84.ec2.internal\" DevicePath \"\""
Sep 11 17:32:07.476439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.476354    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" path="/var/lib/kubelet/pods/2f5ed3f2-791d-428a-9e1a-7d370027a1d6/volumes"
Sep 11 17:32:07.498515 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.498490    2550 generic.go:356] "Generic (PLEG): container finished" podID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerID="aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71" exitCode=0
Sep 11 17:32:07.498627 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.498528    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"
Sep 11 17:32:07.498627 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.498568    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" event={"ID":"e67e0873-ec41-4dbf-92c5-a1f34e56d01c","Type":"ContainerDied","Data":"aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71"}
Sep 11 17:32:07.498627 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.498614    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn" event={"ID":"e67e0873-ec41-4dbf-92c5-a1f34e56d01c","Type":"ContainerDied","Data":"d1ff9ed743eaf8a2a5e0f3dc60c4258b60a0b5cbd7317f3c7fcce92f43fc1182"}
Sep 11 17:32:07.498782 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.498644    2550 scope.go:117] "RemoveContainer" containerID="aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71"
Sep 11 17:32:07.507087 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.507068    2550 scope.go:117] "RemoveContainer" containerID="aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71"
Sep 11 17:32:07.507335 ip-10-0-141-84 kubenswrapper[2550]: E0911 17:32:07.507315    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71\": container with ID starting with aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71 not found: ID does not exist" containerID="aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71"
Sep 11 17:32:07.507400 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.507342    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71"} err="failed to get container status \"aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71\": rpc error: code = NotFound desc = could not find container \"aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71\": container with ID starting with aadce54ba6eb785e0ea308ef175d050f9827b1f9a576dbfee968bc42591bdd71 not found: ID does not exist"
Sep 11 17:32:07.513055 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.513033    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"]
Sep 11 17:32:07.518832 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:07.518813    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-zvxrn"]
Sep 11 17:32:09.476571 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:09.476524    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" path="/var/lib/kubelet/pods/e67e0873-ec41-4dbf-92c5-a1f34e56d01c/volumes"
Sep 11 17:32:35.746689 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:35.746652    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:32:35.800852 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:35.800828    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:32:36.489728 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:36.489695    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:32:36.550609 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:36.550584    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:32:37.281961 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:37.281930    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:32:37.339556 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:37.339519    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:32:38.000967 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:38.000935    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:32:38.057122 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:38.057091    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:32:39.428378 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:39.428335    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:39.439457 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:39.439432    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:39.655044 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:39.655011    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:41.246410 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:41.246374    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:41.269942 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:41.269914    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:41.558293 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:41.558205    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:43.492209 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:43.492178    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:43.503823 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:43.503797    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:43.796300 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:43.796226    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:45.205123 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:45.205096    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:45.216160 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:45.216129    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:45.441692 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:45.441659    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:46.856054 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:46.856021    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:46.867301 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:46.867278    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:47.092798 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:47.092772    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:48.489689 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:48.489657    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:48.502894 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:48.502860    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:48.724763 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:48.724734    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:50.148931 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:50.148899    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:50.160306 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:50.160282    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:50.382583 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:50.382531    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:51.820198 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:51.820169    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:51.831623 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:51.831599    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:52.052427 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:52.052397    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:53.442326 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:53.442295    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:53.454764 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:53.454734    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:53.683804 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:53.683765    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:55.087375 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:55.087340    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:55.099233 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:55.099210    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:55.328213 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:55.328179    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:56.752635 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:56.752603    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:56.763315 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:56.763290    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:56.984234 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:56.984202    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:32:58.405497 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:58.405457    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:32:58.417408 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:58.417389    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:32:58.641993 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:32:58.641964    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:00.040298 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:00.040268    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:00.052202 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:00.052171    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:00.278079 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:00.278035    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:01.704283 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:01.704253    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:01.715745 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:01.715714    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:01.938200 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:01.938173    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:03.361568 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:03.361522    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:03.376888 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:03.376863    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:03.606601 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:03.606565    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:05.019803 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:05.019775    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:05.030433 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:05.030411    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:05.251900 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:05.251872    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:06.640341 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:06.640312    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:06.654009 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:06.653986    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:06.879692 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:06.879663    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:08.272936 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:08.272906    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:08.283604 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:08.283579    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:08.514691 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:08.514663    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:09.919405 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:09.919362    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:09.930118 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:09.930089    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:10.137028 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:10.137004    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:11.531403 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:11.531365    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:11.542566 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:11.542518    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:11.767078 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:11.767052    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:13.170183 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:13.170150    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:13.181578 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:13.181537    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:13.402193 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:13.402163    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:14.812209 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:14.812178    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:14.824932 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:14.824905    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:15.053785 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:15.053755    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:16.447658 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:16.447629    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:16.458721 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:16.458691    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:16.686912 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:16.686880    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:18.074587 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:18.074553    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:18.086887 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:18.086860    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:18.327177 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:18.327097    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:19.727688 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:19.727654    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:19.743754 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:19.743730    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:19.963987 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:19.963958    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:22.067384 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067348    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvg9x/must-gather-tph79"]
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067689    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894bec60-c8d9-455e-88e0-a9dddc70fd23" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067703    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="894bec60-c8d9-455e-88e0-a9dddc70fd23" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067718    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067723    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067730    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067738    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067832    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f5ed3f2-791d-428a-9e1a-7d370027a1d6" containerName="main"
Sep 11 17:33:22.067835 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067848    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67e0873-ec41-4dbf-92c5-a1f34e56d01c" containerName="main"
Sep 11 17:33:22.068281 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.067868    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="894bec60-c8d9-455e-88e0-a9dddc70fd23" containerName="main"
Sep 11 17:33:22.072413 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.072393    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.074261 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.074237    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvg9x\"/\"kube-root-ca.crt\""
Sep 11 17:33:22.074367 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.074237    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvg9x\"/\"openshift-service-ca.crt\""
Sep 11 17:33:22.074367 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.074249    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvg9x\"/\"default-dockercfg-5vsk4\""
Sep 11 17:33:22.079692 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.079476    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/must-gather-tph79"]
Sep 11 17:33:22.129762 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.129737    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98vv\" (UniqueName: \"kubernetes.io/projected/b79bad58-f14c-4e0c-8467-bb47ac642458-kube-api-access-n98vv\") pod \"must-gather-tph79\" (UID: \"b79bad58-f14c-4e0c-8467-bb47ac642458\") " pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.129872 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.129775    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b79bad58-f14c-4e0c-8467-bb47ac642458-must-gather-output\") pod \"must-gather-tph79\" (UID: \"b79bad58-f14c-4e0c-8467-bb47ac642458\") " pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.230848 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.230817    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98vv\" (UniqueName: \"kubernetes.io/projected/b79bad58-f14c-4e0c-8467-bb47ac642458-kube-api-access-n98vv\") pod \"must-gather-tph79\" (UID: \"b79bad58-f14c-4e0c-8467-bb47ac642458\") " pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.230972 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.230856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b79bad58-f14c-4e0c-8467-bb47ac642458-must-gather-output\") pod \"must-gather-tph79\" (UID: \"b79bad58-f14c-4e0c-8467-bb47ac642458\") " pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.231158 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.231141    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b79bad58-f14c-4e0c-8467-bb47ac642458-must-gather-output\") pod \"must-gather-tph79\" (UID: \"b79bad58-f14c-4e0c-8467-bb47ac642458\") " pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.239251 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.239225    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98vv\" (UniqueName: \"kubernetes.io/projected/b79bad58-f14c-4e0c-8467-bb47ac642458-kube-api-access-n98vv\") pod \"must-gather-tph79\" (UID: \"b79bad58-f14c-4e0c-8467-bb47ac642458\") " pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.382354 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.382331    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/must-gather-tph79"
Sep 11 17:33:22.655950 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.655921    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/must-gather-tph79"]
Sep 11 17:33:22.657792 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:33:22.657765    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb79bad58_f14c_4e0c_8467_bb47ac642458.slice/crio-d09dda0ec794a86c2953642a582ca6da9524cb07f9e2d7672425b3db9ce69351 WatchSource:0}: Error finding container d09dda0ec794a86c2953642a582ca6da9524cb07f9e2d7672425b3db9ce69351: Status 404 returned error can't find the container with id d09dda0ec794a86c2953642a582ca6da9524cb07f9e2d7672425b3db9ce69351
Sep 11 17:33:22.659763 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.659743    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:33:22.838233 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:22.838200    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/must-gather-tph79" event={"ID":"b79bad58-f14c-4e0c-8467-bb47ac642458","Type":"ContainerStarted","Data":"d09dda0ec794a86c2953642a582ca6da9524cb07f9e2d7672425b3db9ce69351"}
Sep 11 17:33:23.844910 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:23.844836    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/must-gather-tph79" event={"ID":"b79bad58-f14c-4e0c-8467-bb47ac642458","Type":"ContainerStarted","Data":"5fb4e1429241ab591ef12748c64632f85518470990f68ddaca5a28f16051b4ed"}
Sep 11 17:33:23.844910 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:23.844878    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/must-gather-tph79" event={"ID":"b79bad58-f14c-4e0c-8467-bb47ac642458","Type":"ContainerStarted","Data":"da4418b433911675b4694ad68e30031a1895b2d912c69ace7a72409520c430d3"}
Sep 11 17:33:23.870003 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:23.865086    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tvg9x/must-gather-tph79" podStartSLOduration=0.930539167 podStartE2EDuration="1.865058396s" totalImagesPullingTime="934.519229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:23.861417626 +0000 UTC m=+4578.817465254" watchObservedRunningTime="2026-09-11 17:33:23.865058396 +0000 UTC m=+4578.821106024"
Sep 11 17:33:25.151200 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:25.151170    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-754qk_231ac950-39f0-4afb-a966-3414a5124d8b/global-pull-secret-syncer/0.log"
Sep 11 17:33:25.344180 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:25.344147    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vg278_c5ca5cb4-e2e6-4262-bfbc-d127bd6067ca/konnectivity-agent/0.log"
Sep 11 17:33:25.420366 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:25.420260    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-84.ec2.internal_520fabedc60a64e4fd6648fb59243517/haproxy/0.log"
Sep 11 17:33:30.741513 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:30.741481    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-64hpq_3986030a-ec5f-4a4c-85bc-e49ffd8e2581/manager/0.log"
Sep 11 17:33:30.776002 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:30.775940    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-sdrps_3ca5357c-cbeb-4ea6-b72d-3163db0d2cdb/manager/0.log"
Sep 11 17:33:31.075395 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:31.075265    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-qcd7n_ee02f132-e49c-41cb-bf07-22d37a07bd03/manager/0.log"
Sep 11 17:33:32.391211 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:32.391166    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-ghwt8_076fb9d4-54bd-41e5-9c42-8a9fb362020f/cluster-monitoring-operator/0.log"
Sep 11 17:33:32.669396 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:32.669307    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bzx5d_c5f70df6-5a80-495c-a29e-c18db7697bf3/node-exporter/0.log"
Sep 11 17:33:32.691956 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:32.691924    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bzx5d_c5f70df6-5a80-495c-a29e-c18db7697bf3/kube-rbac-proxy/0.log"
Sep 11 17:33:32.718444 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:32.718414    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bzx5d_c5f70df6-5a80-495c-a29e-c18db7697bf3/init-textfile/0.log"
Sep 11 17:33:35.129528 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.129491    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"]
Sep 11 17:33:35.133882 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.133853    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.140673 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.140647    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"]
Sep 11 17:33:35.242867 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.242832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-lib-modules\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.242867 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.242868    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-podres\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.243072 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.242907    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scd84\" (UniqueName: \"kubernetes.io/projected/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-kube-api-access-scd84\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.243072 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.242945    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-sys\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.243072 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.243021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-proc\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.344230 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.344191    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-proc\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.344694 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.344667    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-lib-modules\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.344840 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.344824    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-podres\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.344956 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.344937    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-lib-modules\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.345025 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.344392    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-proc\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.345089 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.345056    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-podres\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.345178 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.344949    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scd84\" (UniqueName: \"kubernetes.io/projected/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-kube-api-access-scd84\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.345318 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.345303    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-sys\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.345499 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.345480    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-sys\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.352846 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.352823    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scd84\" (UniqueName: \"kubernetes.io/projected/b99a10ec-3fcf-46ee-86a1-9419a4b519f2-kube-api-access-scd84\") pod \"perf-node-gather-daemonset-f9zzj\" (UID: \"b99a10ec-3fcf-46ee-86a1-9419a4b519f2\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.445351 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.445275    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.450304 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.450276    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-qtjs4_4d6a0e34-05fa-4bd7-beba-1caf4426e69f/download-server/0.log"
Sep 11 17:33:35.740731 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.740698    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"]
Sep 11 17:33:35.748365 ip-10-0-141-84 kubenswrapper[2550]: W0911 17:33:35.748332    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb99a10ec_3fcf_46ee_86a1_9419a4b519f2.slice/crio-5b55aba8b3962c2ee994f2c9afd9204df28b43479cb67709715cb07007df97fb WatchSource:0}: Error finding container 5b55aba8b3962c2ee994f2c9afd9204df28b43479cb67709715cb07007df97fb: Status 404 returned error can't find the container with id 5b55aba8b3962c2ee994f2c9afd9204df28b43479cb67709715cb07007df97fb
Sep 11 17:33:35.933847 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.933801    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj" event={"ID":"b99a10ec-3fcf-46ee-86a1-9419a4b519f2","Type":"ContainerStarted","Data":"884748a5d67163d0a0ce94836c954bb9701150eafa7a323d8811118de2572f7e"}
Sep 11 17:33:35.934043 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.933874    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj" event={"ID":"b99a10ec-3fcf-46ee-86a1-9419a4b519f2","Type":"ContainerStarted","Data":"5b55aba8b3962c2ee994f2c9afd9204df28b43479cb67709715cb07007df97fb"}
Sep 11 17:33:35.934133 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.934111    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:35.949663 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:35.949606    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj" podStartSLOduration=0.949587145 podStartE2EDuration="949.587145ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:35.947927126 +0000 UTC m=+4590.903974755" watchObservedRunningTime="2026-09-11 17:33:35.949587145 +0000 UTC m=+4590.905634774"
Sep 11 17:33:37.082401 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:37.082366    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgtfg_61c0e3ee-7239-48a6-8823-82be2e360cae/dns/0.log"
Sep 11 17:33:37.102046 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:37.102004    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pgtfg_61c0e3ee-7239-48a6-8823-82be2e360cae/kube-rbac-proxy/0.log"
Sep 11 17:33:37.171396 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:37.171368    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jx4cp_b902a3c9-3153-4972-a0d5-eff6e6aec791/dns-node-resolver/0.log"
Sep 11 17:33:37.767296 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:37.767264    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wbwf_3425dd2f-36b9-4a54-bdcb-1989ae091941/node-ca/0.log"
Sep 11 17:33:38.833286 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:38.833220    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:33:38.908263 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:38.908226    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:33:39.459090 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:39.459058    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdntq_379fc8b9-b464-4ce9-bf17-882739302bdf/serve-healthcheck-canary/0.log"
Sep 11 17:33:39.941687 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:39.941653    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9q95b_9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1/kube-rbac-proxy/0.log"
Sep 11 17:33:39.961615 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:39.961587    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9q95b_9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1/exporter/0.log"
Sep 11 17:33:39.982039 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:39.982009    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9q95b_9e2e4cf6-8d6a-4f6f-a59e-d8578917d1b1/extractor/0.log"
Sep 11 17:33:41.950776 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:41.950740    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-f9zzj"
Sep 11 17:33:43.660462 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:43.660426    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-656b75fc8d-p8vmj_9b471abb-626e-4430-b15c-57694b24b497/manager/0.log"
Sep 11 17:33:44.338992 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:44.338911    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-mmb5z_10b8e008-b3b9-48e3-a438-2df7cd58d09b/server/0.log"
Sep 11 17:33:44.670164 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:44.670114    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:33:44.739883 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:44.739856    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:33:45.608723 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:45.608694    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-hdjb4_76bee7ac-7333-47d7-88c3-2fc651e5c92e/manager/0.log"
Sep 11 17:33:45.761147 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:45.761118    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-lghrk_b20cbee5-2792-4ed2-a389-5e482ad054fb/seaweedfs/0.log"
Sep 11 17:33:45.991755 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:45.991671    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8f972_d2c4af5c-e523-46ac-a53b-93b38efccf5d/discovery/0.log"
Sep 11 17:33:46.055414 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:46.055385    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78b74d9c58-fgt69_ba8f0c3a-d449-423a-94ff-5b2ca617c983/router/0.log"
Sep 11 17:33:50.323503 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:50.323476    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-p65wk_7070d490-016f-478a-a093-778cc0eb45bd/migrator/0.log"
Sep 11 17:33:50.344984 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:50.344951    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-p65wk_7070d490-016f-478a-a093-778cc0eb45bd/graceful-termination/0.log"
Sep 11 17:33:51.641156 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.641126    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/kube-multus-additional-cni-plugins/0.log"
Sep 11 17:33:51.661174 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.661149    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/egress-router-binary-copy/0.log"
Sep 11 17:33:51.681068 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.681043    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/cni-plugins/0.log"
Sep 11 17:33:51.703187 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.703165    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/bond-cni-plugin/0.log"
Sep 11 17:33:51.732306 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.732286    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/routeoverride-cni/0.log"
Sep 11 17:33:51.752002 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.751985    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/whereabouts-cni-bincopy/0.log"
Sep 11 17:33:51.772580 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:51.772558    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dlccg_b1a4f2f1-7987-46fa-ae27-600b250b0d12/whereabouts-cni/0.log"
Sep 11 17:33:52.236439 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:52.236406    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qk8cl_3306db7f-f969-4cac-a413-5a9400e48e99/kube-multus/0.log"
Sep 11 17:33:52.258328 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:52.258303    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vcxfz_dac1c7b7-7288-434a-8196-cd1a17d2f0b8/network-metrics-daemon/0.log"
Sep 11 17:33:52.276188 ip-10-0-141-84 kubenswrapper[2550]: I0911 17:33:52.276166    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vcxfz_dac1c7b7-7288-434a-8196-cd1a17d2f0b8/kube-rbac-proxy/0.log"