Sep 17 17:34:33.686909 ip-10-0-141-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:36.856449 ip-10-0-141-122 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:36.856449 ip-10-0-141-122 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:36.856449 ip-10-0-141-122 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:36.856449 ip-10-0-141-122 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:36.856449 ip-10-0-141-122 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:36.857351 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.856516    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:36.859346 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859331    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:36.859346 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859346    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859350    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859354    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859356    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859365    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859368    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859370    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859373    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859376    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859378    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859381    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859384    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859387    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859390    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859392    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859395    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859397    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859400    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859403    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859405    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:36.859412 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859408    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859410    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859413    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859416    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859419    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859421    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859424    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859427    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859429    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859432    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859435    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859438    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859440    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859443    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859445    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859448    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859451    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859453    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859456    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859458    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:36.859910 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859461    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859464    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859467    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859469    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859472    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859474    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859477    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859480    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859482    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859485    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859487    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859490    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859493    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859495    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859498    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859501    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859504    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859507    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859509    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859512    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:36.860399 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859514    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859517    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859520    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859523    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859526    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859528    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859530    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859533    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859536    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859538    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859541    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859543    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859546    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859549    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859551    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859554    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859557    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859559    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859563    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859565    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:36.860900 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859568    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859572    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859576    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859580    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859584    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859588    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859591    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859594    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859597    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859600    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859602    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859606    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859610    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859613    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859616    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859619    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859622    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.859624    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860313    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:36.861408 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860320    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860323    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860327    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860329    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860332    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860335    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860338    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860341    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860345    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860347    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860350    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860353    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860356    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860359    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860362    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860364    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860368    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860371    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860374    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860376    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:36.861921 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860379    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860381    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860384    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860387    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860390    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860394    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860398    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860401    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860404    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860406    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860411    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860417    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860420    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860424    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860427    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860430    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860433    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860436    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:36.862411 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860438    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860441    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860444    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860446    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860449    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860452    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860454    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860457    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860459    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860461    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860464    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860468    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860470    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860473    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860475    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860478    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860481    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860483    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860486    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860488    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:36.862876 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860491    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860493    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860496    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860498    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860501    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860505    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860508    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860510    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860513    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860515    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860518    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860520    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860523    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860525    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860528    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860530    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860534    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860536    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860539    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860542    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:36.863364 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860544    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860547    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860550    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860553    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860555    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860558    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860560    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860562    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860565    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860567    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860571    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860574    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860576    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860579    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860581    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860584    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860587    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860590    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860593    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860595    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:36.863862 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860958    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860963    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860966    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860969    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860972    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860975    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860977    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860980    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860983    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860986    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860988    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860991    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860994    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860996    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.860999    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861002    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861005    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861008    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861010    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861013    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:36.864354 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861016    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861019    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861021    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861024    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861028    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861030    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861033    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861035    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861038    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861041    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861044    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861047    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861051    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861054    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861057    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861060    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861063    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861066    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861069    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:36.864853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861071    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861074    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861077    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861080    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861082    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861085    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861087    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861092    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861095    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861098    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861100    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861103    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861106    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861108    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861111    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861113    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861116    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861119    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861122    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861125    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:36.865327 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861128    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861131    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861134    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861136    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861139    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861142    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861145    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861147    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861150    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861152    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861155    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861157    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861160    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861163    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861165    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861168    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861170    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861173    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861175    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861179    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:36.865841 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861182    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861184    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861187    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861190    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861192    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861195    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861198    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861200    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861203    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861205    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861209    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861212    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861215    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861218    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861223    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861225    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861228    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861232    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861234    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:36.866341 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861237    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861309    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861344    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861350    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861354    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861358    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861361    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861365    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861369    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861372    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861376    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861379    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861386    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861389    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861392    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861395    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861398    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861401    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861404    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861407    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861410    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861414    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861417    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861420    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:36.866866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861424    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861427    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861433    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861436    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861439    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861442    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861445    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861449    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861452    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861456    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861459    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861462    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861465    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861467    2561 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861470    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861475    2561 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861478    2561 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861481    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861484    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861486    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861491    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861494    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861497    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861500    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861503    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:36.867442 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861506    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861509    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861512    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861515    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861518    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861521    2561 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861524    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861527    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861530    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861535    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861538    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861541    2561 flags.go:64] FLAG: --help="false"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861544    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-141-122.ec2.internal"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861547    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861550    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861553    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861556    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861559    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861562    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861565    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861568    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861570    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861573    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861576    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:36.868061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861579    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861582    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861585    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861589    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861592    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861595    2561 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861598    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861601    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861603    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861609    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861612    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861614    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861617    2561 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861620    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861623    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861626    2561 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861645    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861658    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861662    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861666    2561 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861668    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861672    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861674    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861677    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861681    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:36.868683 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861684    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861687    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861694    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861697    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861700    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861703    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861706    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861712    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861715    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861718    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861721    2561 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861727    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861730    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07805282c49ee5341"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861733    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861736    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861739    2561 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861742    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861745    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861748    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861751    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861754    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861758    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861761    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861763    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861768    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:36.869297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861771    2561 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861773    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861776    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861779    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861782    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861785    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861788    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861792    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861795    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861798    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861801    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861804    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861807    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861810    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861813    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861816    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861821    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861824    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861828    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861832    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861835    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861838    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861841    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861844    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861846    2561 flags.go:64] FLAG: --v="2"
Sep 17 17:34:36.869913 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861850    2561 flags.go:64] FLAG: --version="false"
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861854    2561 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861857    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.861861    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861971    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861975    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861980    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861983    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861988    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861991    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861995    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.861998    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862001    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862004    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862007    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862011    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862014    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862016    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862027    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:36.870556 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862030    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862033    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862035    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862038    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862041    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862043    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862047    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862050    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862052    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862055    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862058    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862060    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862063    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862067    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862070    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862073    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862076    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862078    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862081    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:36.871129 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862085    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862087    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862091    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862095    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862098    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862100    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862103    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862106    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862109    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862112    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862116    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862118    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862121    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862123    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862126    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862128    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862131    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862134    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862136    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862139    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:36.871948 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862142    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862144    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862147    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862149    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862152    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862154    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862157    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862160    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862162    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862165    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862168    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862170    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862174    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862177    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862179    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862182    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862184    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862187    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862190    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862192    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:36.872529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862195    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862198    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862200    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862203    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862206    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862208    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862211    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862213    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862216    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862218    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862220    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862223    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862226    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862228    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862230    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862233    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862235    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862238    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862240    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862243    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:36.873031 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862245    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862249    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862251    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862254    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.862257    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.863014    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.870080    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.870094    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.870113    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.870119    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870199    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870205    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870208    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870211    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870214    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:36.873534 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870223    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870226    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870229    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870231    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870234    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870237    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870239    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870242    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870245    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870248    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870250    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870253    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870256    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870258    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870261    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870263    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870266    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870269    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870272    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870274    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:36.873941 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870277    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870280    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870282    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870285    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870288    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870291    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870293    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870296    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870299    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870301    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870304    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870307    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870310    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870321    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870324    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870327    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870331    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870335    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870338    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:36.874451 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870341    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870344    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870347    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870350    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870353    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870355    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870358    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870361    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870363    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870366    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870368    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870371    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870373    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870376    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870378    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870381    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870384    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870387    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870390    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870392    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:36.875041 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870395    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870399    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870402    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870405    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870408    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870410    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870413    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870422    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870425    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870428    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870431    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870433    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870436    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870439    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870441    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870444    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870447    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870449    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870452    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:36.875539 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870454    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870457    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870459    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870462    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870464    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870467    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870470    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870472    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870475    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870479    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870483    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870486    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870489    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870491    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870494    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:36.876013 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870496    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.870501    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870693    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870699    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870702    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870705    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870713    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870716    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870719    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870722    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870724    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870727    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870730    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870733    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870735    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:36.876416 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870738    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870740    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870744    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870748    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870751    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870754    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870756    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870759    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870763    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870766    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870769    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870772    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870776    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870779    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870782    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870785    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870787    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870790    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:36.876813 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870794    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870797    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870800    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870803    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870805    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870808    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.870858    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871020    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871026    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871030    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871034    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871038    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871042    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871047    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871052    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871057    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871062    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871069    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871076    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871083    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:36.877243 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871095    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871102    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871109    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871116    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871124    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871131    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871139    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871146    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871153    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871160    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871169    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871177    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871189    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871196    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871203    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871210    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871218    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871226    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871233    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871272    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:36.877745 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871280    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871287    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871295    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871302    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871316    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871323    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871330    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871338    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871344    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871352    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871359    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871365    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871373    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871380    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871386    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871393    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871399    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871411    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871453    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871459    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:36.878220 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871465    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871472    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871480    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871486    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871492    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871499    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871505    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:36.871512    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.871521    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.871897    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.873955    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.874837    2561 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.874934    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:36.878800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.874964    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:36.879140 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.877131    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:36.879140 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.878276    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:36.886827 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.886812    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:36.888258 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.888236    2561 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:36.888984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.888969    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:36.891939 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.891922    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c1a400f5-de87-4bdb-9ce0-55a4c8e5ae05:/dev/nvme0n1p4 cf8cc585-92f9-4bc6-a829-4632a2c5d407:/dev/nvme0n1p3]
Sep 17 17:34:36.891995 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.891939    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:36.897474 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.897376    2561 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:36.896901374 +0000 UTC m=+1.856786635 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100073 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2302ee781318ae3924f06e4f0cbc8b SystemUUID:ec2302ee-7813-18ae-3924-f06e4f0cbc8b BootID:812330fe-b2b4-4d7f-b69d-8813c6c3cd0e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:f0:f2:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:f0:f2:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:74:6b:ab:db:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:36.897474 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.897470    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:36.897576 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.897567    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:36.899033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899009    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:36.899161 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899037    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:36.899860 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899851    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:36.899891 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899862    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:36.899891 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899870    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:36.899891 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899880    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:36.899974 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.899944    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:36.903775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903764    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:36.903878 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903870    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:36.903910 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903901    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:36.903910 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903910    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:36.903965 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903923    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:36.903965 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903931    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:36.903965 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.903939    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:36.904246 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.904234    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:36.904327 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.904255    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:36.907898 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.907878    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:36.910712 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.910697    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:36.910789 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.910727    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:36.911948 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911936    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:36.911984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911954    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:36.911984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911960    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:36.911984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911965    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:36.911984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911973    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:36.911984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911978    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911991    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.911996    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.912002    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.912007    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.912016    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.912020    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.912041    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:36.912122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.912046    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:36.915519 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.915501    2561 server.go:1268] "Started kubelet"
Sep 17 17:34:36.915609 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.915564    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:36.915692 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.915574    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:36.915692 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.915666    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:36.916432 ip-10-0-141-122 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:36.916854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.916673    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:36.917361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.917347    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:36.918410 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.918377    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:36.918484 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.918415    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:36.918484 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.918430    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:36.919707 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.919690    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:36.923494 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.923474    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:36.923494 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.923491    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:36.923818 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.923796    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:36.924123 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.923906    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:36.924123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924053    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:36.924123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924103    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:36.924123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924115    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:36.924328 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924236    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:36.924328 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924251    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:36.924674 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924657    2561 factory.go:153] Registering CRI-O factory
Sep 17 17:34:36.924674 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924674    2561 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:36.924775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924725    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:36.924775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924733    2561 factory.go:55] Registering systemd factory
Sep 17 17:34:36.924775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924741    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:36.924775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924754    2561 factory.go:103] Registering Raw factory
Sep 17 17:34:36.924775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.924764    2561 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:36.925331 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.925218    2561 manager.go:349] Starting recovery of all containers
Sep 17 17:34:36.932139 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.932114    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:34:36.932283 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.932255    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:34:36.943939 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.943919    2561 manager.go:354] Recovery completed
Sep 17 17:34:36.948505 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.948491    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:36.950120 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950103    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:36.950194 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950134    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:36.950194 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950148    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:36.950678 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950665    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:36.950678 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950675    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:36.950781 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950677    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:36.950781 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.950752    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:36.951395 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.951381    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:36.951449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.951403    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:36.951449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.951413    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:36.952964 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.952951    2561 policy_none.go:49] "None policy: Start"
Sep 17 17:34:36.953022 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.952967    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:36.953022 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.952978    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:36.954981 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.954970    2561 policy_none.go:47] "Start"
Sep 17 17:34:36.962702 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.961852    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-122.ec2.internal.18d62c2dbef8bf88  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-122.ec2.internal,UID:ip-10-0-141-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-122.ec2.internal,},FirstTimestamp:2026-09-17 17:34:36.915474312 +0000 UTC m=+1.875359572,LastTimestamp:2026-09-17 17:34:36.915474312 +0000 UTC m=+1.875359572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-122.ec2.internal,}"
Sep 17 17:34:36.978076 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.977982    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-122.ec2.internal.18d62c2dc1096179  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-122.ec2.internal,UID:ip-10-0-141-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-122.ec2.internal,},FirstTimestamp:2026-09-17 17:34:36.950118777 +0000 UTC m=+1.910004045,LastTimestamp:2026-09-17 17:34:36.950118777 +0000 UTC m=+1.910004045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-122.ec2.internal,}"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986150    2561 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.986210    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986224    2561 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986432    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986450    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986527    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986606    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.986616    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.987353    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.987394    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.988893    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phxcn"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:36.992947    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-122.ec2.internal.18d62c2dc109b883  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-122.ec2.internal,UID:ip-10-0-141-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-122.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-122.ec2.internal,},FirstTimestamp:2026-09-17 17:34:36.950141059 +0000 UTC m=+1.910026322,LastTimestamp:2026-09-17 17:34:36.950141059 +0000 UTC m=+1.910026322,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-122.ec2.internal,}"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:36.999428    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phxcn"
Sep 17 17:34:37.005047 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.002134    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-122.ec2.internal.18d62c2dc109ec87  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-122.ec2.internal,UID:ip-10-0-141-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-122.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-122.ec2.internal,},FirstTimestamp:2026-09-17 17:34:36.950154375 +0000 UTC m=+1.910039637,LastTimestamp:2026-09-17 17:34:36.950154375 +0000 UTC m=+1.910039637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-122.ec2.internal,}"
Sep 17 17:34:37.012609 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.012584    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:37.013794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.013778    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:37.013866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.013822    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:37.013866 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.013841    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:37.013955 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.013888    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:37.018083 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.018064    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:37.087431 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.087413    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:37.088271 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.088257    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:37.088351 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.088288    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:37.088351 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.088299    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:37.088351 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.088320    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.102767 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.102749    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.102815 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.102768    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-122.ec2.internal\": node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.114407 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.114360    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"]
Sep 17 17:34:37.114470 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.114417    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:37.115188 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.115174    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:37.115258 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.115200    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:37.115258 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.115212    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:37.117466 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.117451    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:37.117577 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.117563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.117623 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.117595    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:37.120065 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.120049    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:37.120065 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.120059    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:37.120202 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.120076    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:37.120202 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.120078    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:37.120202 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.120125    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:37.120202 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.120091    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:37.121867 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.121848    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.122360 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.122345    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.122429 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.122369    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:37.123138 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.123124    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:37.123212 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.123148    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:37.123212 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.123158    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:37.124950 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.124928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27263fbfefb9479050311e268f48deef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal\" (UID: \"27263fbfefb9479050311e268f48deef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.125037 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.124963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27263fbfefb9479050311e268f48deef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal\" (UID: \"27263fbfefb9479050311e268f48deef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.125037 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.124987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15339a5bbb7d25ed5fa56da3efddf311-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.ec2.internal\" (UID: \"15339a5bbb7d25ed5fa56da3efddf311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.145147 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.145125    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-122.ec2.internal\" not found" node="ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.149365 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.149350    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-122.ec2.internal\" not found" node="ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.222117 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.222102    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.225374 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.225359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27263fbfefb9479050311e268f48deef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal\" (UID: \"27263fbfefb9479050311e268f48deef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.225423 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.225382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27263fbfefb9479050311e268f48deef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal\" (UID: \"27263fbfefb9479050311e268f48deef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.225423 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.225399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15339a5bbb7d25ed5fa56da3efddf311-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.ec2.internal\" (UID: \"15339a5bbb7d25ed5fa56da3efddf311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.225423 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.225402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27263fbfefb9479050311e268f48deef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal\" (UID: \"27263fbfefb9479050311e268f48deef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.225511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.225420    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/15339a5bbb7d25ed5fa56da3efddf311-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.ec2.internal\" (UID: \"15339a5bbb7d25ed5fa56da3efddf311\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.225511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.225441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27263fbfefb9479050311e268f48deef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal\" (UID: \"27263fbfefb9479050311e268f48deef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.323071 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.323036    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.423767 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.423707    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.450089 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.450072    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.453228 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.453213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:37.524545 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.524519    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.624948 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.624928    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.725424 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.725362    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.825793 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.825766    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:37.875082 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.875053    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:37.875652 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.875190    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:37.914689 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:37.914669    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:37.925914 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:37.925895    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:38.005087 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.005025    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:36 +0000 UTC" deadline="2028-02-29 00:13:01.755917175 +0000 UTC"
Sep 17 17:34:38.005087 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.005051    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12702h38m23.750868425s"
Sep 17 17:34:38.026741 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.026690    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:38.026741 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.026704    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:38.027929 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.027913    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:38.028031 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.027946    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:38.028031 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.027961    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:38.126916 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.126888    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:38.171648 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:38.171603    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27263fbfefb9479050311e268f48deef.slice/crio-04f91b9baf6bff48f08f1d205c5aeec74947dc0cc686848889485b5dea271e08 WatchSource:0}: Error finding container 04f91b9baf6bff48f08f1d205c5aeec74947dc0cc686848889485b5dea271e08: Status 404 returned error can't find the container with id 04f91b9baf6bff48f08f1d205c5aeec74947dc0cc686848889485b5dea271e08
Sep 17 17:34:38.171875 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:38.171844    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15339a5bbb7d25ed5fa56da3efddf311.slice/crio-38888b84e7122c0aaf7b53c391cff90b94aead16960300e77d01558cce8a82df WatchSource:0}: Error finding container 38888b84e7122c0aaf7b53c391cff90b94aead16960300e77d01558cce8a82df: Status 404 returned error can't find the container with id 38888b84e7122c0aaf7b53c391cff90b94aead16960300e77d01558cce8a82df
Sep 17 17:34:38.177921 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.177906    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:38.181876 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.181858    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:38.227350 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.227326    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:38.327834 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.327787    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-122.ec2.internal\" not found"
Sep 17 17:34:38.415355 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.415332    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:38.424451 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.424432    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:38.437962 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.437945    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:38.438781 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.438769    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal"
Sep 17 17:34:38.447116 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.447100    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:38.905584 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.905556    2561 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:38.913966 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.913939    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:38.914351 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.914326    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-kp8xn","openshift-multus/network-metrics-daemon-qdh2w","kube-system/konnectivity-agent-v5vhn","kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh","openshift-cluster-node-tuning-operator/tuned-q8bjh","openshift-dns/node-resolver-7qdqz","openshift-network-diagnostics/network-check-target-v9xw9","openshift-network-operator/iptables-alerter-66qpz","openshift-ovn-kubernetes/ovnkube-node-pddbm","kube-system/global-pull-secret-syncer-d6dz7","openshift-image-registry/node-ca-4jfq7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal","openshift-multus/multus-8lsrh"]
Sep 17 17:34:38.917029 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.917008    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.919378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.919283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:38.919378 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.919362    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:38.921472 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.921453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:38.922840 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.922677    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:38.922840 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.922786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:38.923356 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.923216    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:38.923356 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.923273    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.923724 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.923706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.923807 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.923777    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:38.924425 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.924374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8nck5\""
Sep 17 17:34:38.924524 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.924454    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.924623 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.924600    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:38.924750 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.924707    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:38.925225 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.925178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:38.926392 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.926376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.928190 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.928148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k4dp7\""
Sep 17 17:34:38.928467 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.928450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:38.930060 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.930045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:38.930147 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.930068    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.930734 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.930716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:38.932686 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.932664    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.932775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.932703    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.933131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.933113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:38.933206 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.933180    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:38.934247 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-kubelet-config\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:38.934346 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a1efed-a8d2-4761-bc6a-c180cda036d0-cni-binary-copy\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.934346 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-k8s-cni-cncf-io\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.934346 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-multus-certs\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.934346 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-sys-fs\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.934578 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-kubelet\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.934578 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-systemd-units\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.934578 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-ovn\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.934578 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934498    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-cni-bin\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.934578 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-etc-selinux\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.934578 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.934545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-var-lib-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.935032 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-node-log\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.935105 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e2c70-905d-42e5-a325-bb57bdb2edcd-konnectivity-ca\") pod \"konnectivity-agent-v5vhn\" (UID: \"7d1e2c70-905d-42e5-a325-bb57bdb2edcd\") " pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:38.935178 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fm4\" (UniqueName: \"kubernetes.io/projected/52a1efed-a8d2-4761-bc6a-c180cda036d0-kube-api-access-m7fm4\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.935178 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-registration-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.935178 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935150    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:38.935335 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-cni-bin\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.935391 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b754174e-ba79-49f2-ae27-2039f91958b4-ovn-node-metrics-cert\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.935391 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-dbus\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:38.935508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d1e2c70-905d-42e5-a325-bb57bdb2edcd-agent-certs\") pod \"konnectivity-agent-v5vhn\" (UID: \"7d1e2c70-905d-42e5-a325-bb57bdb2edcd\") " pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:38.935508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-cnibin\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.935508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-kubelet\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.935508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-conf-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.935508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935506    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.935779 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-slash\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.935779 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-cni-netd\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.935779 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-cni-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.935994 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.935973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-netns\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.936056 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-cni-multus\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.936106 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-socket-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.936175 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzxn\" (UniqueName: \"kubernetes.io/projected/ee9905e7-955d-42f6-9816-a06d3b658ef4-kube-api-access-rfzxn\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.936175 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936276 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936329 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-ovnkube-script-lib\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936379 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:38.936432 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-socket-dir-parent\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.936432 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936397    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:38.936432 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-daemon-config\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.936570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-etc-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-ovnkube-config\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l924\" (UniqueName: \"kubernetes.io/projected/b754174e-ba79-49f2-ae27-2039f91958b4-kube-api-access-8l924\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936745 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-system-cni-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.936745 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936665    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.936745 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-hostroot\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.936925 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-log-socket\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936925 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-systemd\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936925 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.936925 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-etc-kubernetes\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:38.937131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-device-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:38.937131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.936986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-run-netns\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.937131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.937009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-env-overrides\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:38.937131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.937019    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lpklv\""
Sep 17 17:34:38.937472 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.937450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:38.937540 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.937489    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.937540 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:38.937528    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:38.940219 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.940199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:38.943485 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.943465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:38.945485 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.945465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:38.951924 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.951906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zl5dp\""
Sep 17 17:34:38.952030 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.951907    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.958767 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.958747    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4jdj4\""
Sep 17 17:34:38.958839 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.958788    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.958839 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.958821    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k2ldr\""
Sep 17 17:34:38.958988 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.958968    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:38.959038 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.958986    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.959038 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.958989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:38.959100 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959052    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.959137 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959100    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.959137 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959112    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:38.959204 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959192    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7r8f\""
Sep 17 17:34:38.959235 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959205    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2rwpn\""
Sep 17 17:34:38.959235 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959215    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:38.959404 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.959387    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:38.986584 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.986557    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:38.989404 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.989385    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gg4fk\""
Sep 17 17:34:38.996534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.996514    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:38.996612 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:38.996593    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6khmr"
Sep 17 17:34:39.017572 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.017546    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6khmr"
Sep 17 17:34:39.019008 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.018964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal" event={"ID":"15339a5bbb7d25ed5fa56da3efddf311","Type":"ContainerStarted","Data":"38888b84e7122c0aaf7b53c391cff90b94aead16960300e77d01558cce8a82df"}
Sep 17 17:34:39.020300 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.020279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal" event={"ID":"27263fbfefb9479050311e268f48deef","Type":"ContainerStarted","Data":"04f91b9baf6bff48f08f1d205c5aeec74947dc0cc686848889485b5dea271e08"}
Sep 17 17:34:39.025228 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.025209    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:39.037144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-tmp\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.037248 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-run-netns\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.037248 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-env-overrides\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.037248 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-run-netns\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.037248 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-kubelet-config\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-k8s-cni-cncf-io\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-multus-certs\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-sys-fs\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-kubelet-config\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92ad8a17-1941-460c-ba02-ba8d8237f79d-host-slash\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-modprobe-d\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-systemd-units\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-ovn\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-etc-selinux\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjlz\" (UniqueName: \"kubernetes.io/projected/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-kube-api-access-5kjlz\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:39.037520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27603f15-420b-4c05-aaa8-fea5ecb33710-serviceca\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-var-lib-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-systemd-units\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037608    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-ovn\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037669    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-etc-selinux\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-var-lib-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-node-log\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-env-overrides\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-multus-certs\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fm4\" (UniqueName: \"kubernetes.io/projected/52a1efed-a8d2-4761-bc6a-c180cda036d0-kube-api-access-m7fm4\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037755    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-sys-fs\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-socket-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-node-log\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-k8s-cni-cncf-io\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-cni-bin\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b754174e-ba79-49f2-ae27-2039f91958b4-ovn-node-metrics-cert\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-dbus\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-cnibin\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.038135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-kubelet\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037986    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-socket-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.037998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-cni-bin\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-kubernetes\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-dbus\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysctl-conf\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-cni-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038078    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-kubelet\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-cnibin\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-cni-multus\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-cni-multus\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysconfig\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-tuned\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-cnibin\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-cni-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038349    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27603f15-420b-4c05-aaa8-fea5ecb33710-host\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvx4\" (UniqueName: \"kubernetes.io/projected/27603f15-420b-4c05-aaa8-fea5ecb33710-kube-api-access-txvx4\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.039144 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-sys\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.038530    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxmx\" (UniqueName: \"kubernetes.io/projected/3e13d86a-6c05-4f96-a87a-a075c01901ac-kube-api-access-hwxmx\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-system-cni-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.038644    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.538588794 +0000 UTC m=+4.498474061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-system-cni-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-hostroot\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-lib-modules\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-hostroot\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-log-socket\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh82f\" (UniqueName: \"kubernetes.io/projected/92ad8a17-1941-460c-ba02-ba8d8237f79d-kube-api-access-wh82f\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-systemd\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-log-socket\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-hosts-file\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.040171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-tmp-dir\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038875    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a1efed-a8d2-4761-bc6a-c180cda036d0-cni-binary-copy\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysctl-d\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltx5\" (UniqueName: \"kubernetes.io/projected/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-kube-api-access-4ltx5\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038969    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-kubelet\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.038995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-cni-bin\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039062    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-kubelet\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-var-lib-cni-bin\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-var-lib-kubelet\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e2c70-905d-42e5-a325-bb57bdb2edcd-konnectivity-ca\") pod \"konnectivity-agent-v5vhn\" (UID: \"7d1e2c70-905d-42e5-a325-bb57bdb2edcd\") " pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-registration-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92ad8a17-1941-460c-ba02-ba8d8237f79d-iptables-alerter-script\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzxn\" (UniqueName: \"kubernetes.io/projected/ee9905e7-955d-42f6-9816-a06d3b658ef4-kube-api-access-rfzxn\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d1e2c70-905d-42e5-a325-bb57bdb2edcd-agent-certs\") pod \"konnectivity-agent-v5vhn\" (UID: \"7d1e2c70-905d-42e5-a325-bb57bdb2edcd\") " pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-conf-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.040918 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-host\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-slash\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-cni-netd\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039277    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-netns\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-run\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-system-cni-dir\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-ovnkube-script-lib\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-socket-dir-parent\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-daemon-config\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-etc-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52a1efed-a8d2-4761-bc6a-c180cda036d0-cni-binary-copy\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-ovnkube-config\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l924\" (UniqueName: \"kubernetes.io/projected/b754174e-ba79-49f2-ae27-2039f91958b4-kube-api-access-8l924\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bx4\" (UniqueName: \"kubernetes.io/projected/589f88ca-b342-488a-9e88-4c60c80dd6bb-kube-api-access-p5bx4\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:39.041801 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-cni-netd\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039767    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-ovnkube-config\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-host-run-netns\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039839    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-etc-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-socket-dir-parent\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-conf-dir\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.039978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-registration-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-systemd\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-host-slash\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040164    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-etc-kubernetes\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-device-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d1e2c70-905d-42e5-a325-bb57bdb2edcd-konnectivity-ca\") pod \"konnectivity-agent-v5vhn\" (UID: \"7d1e2c70-905d-42e5-a325-bb57bdb2edcd\") " pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:39.042794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee9905e7-955d-42f6-9816-a06d3b658ef4-device-dir\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.043702 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-systemd\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.043702 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b754174e-ba79-49f2-ae27-2039f91958b4-ovnkube-script-lib\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.043702 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040407    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b754174e-ba79-49f2-ae27-2039f91958b4-run-openvswitch\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.043702 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52a1efed-a8d2-4761-bc6a-c180cda036d0-etc-kubernetes\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.043702 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.040410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/52a1efed-a8d2-4761-bc6a-c180cda036d0-multus-daemon-config\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.043972 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.043769    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d1e2c70-905d-42e5-a325-bb57bdb2edcd-agent-certs\") pod \"konnectivity-agent-v5vhn\" (UID: \"7d1e2c70-905d-42e5-a325-bb57bdb2edcd\") " pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:39.044790 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.044771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b754174e-ba79-49f2-ae27-2039f91958b4-ovn-node-metrics-cert\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.052702 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.052657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzxn\" (UniqueName: \"kubernetes.io/projected/ee9905e7-955d-42f6-9816-a06d3b658ef4-kube-api-access-rfzxn\") pod \"aws-ebs-csi-driver-node-828fh\" (UID: \"ee9905e7-955d-42f6-9816-a06d3b658ef4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.053036 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.053008    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fm4\" (UniqueName: \"kubernetes.io/projected/52a1efed-a8d2-4761-bc6a-c180cda036d0-kube-api-access-m7fm4\") pod \"multus-8lsrh\" (UID: \"52a1efed-a8d2-4761-bc6a-c180cda036d0\") " pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.054208 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.054168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l924\" (UniqueName: \"kubernetes.io/projected/b754174e-ba79-49f2-ae27-2039f91958b4-kube-api-access-8l924\") pod \"ovnkube-node-pddbm\" (UID: \"b754174e-ba79-49f2-ae27-2039f91958b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.141354 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-tmp\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141354 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92ad8a17-1941-460c-ba02-ba8d8237f79d-host-slash\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141372    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-modprobe-d\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjlz\" (UniqueName: \"kubernetes.io/projected/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-kube-api-access-5kjlz\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92ad8a17-1941-460c-ba02-ba8d8237f79d-host-slash\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27603f15-420b-4c05-aaa8-fea5ecb33710-serviceca\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-modprobe-d\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141556 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-kubernetes\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysctl-conf\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.141616    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-kubernetes\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.141684    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.641666397 +0000 UTC m=+4.601551652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysconfig\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysctl-conf\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-tuned\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141753    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-cnibin\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27603f15-420b-4c05-aaa8-fea5ecb33710-host\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysconfig\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-cnibin\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txvx4\" (UniqueName: \"kubernetes.io/projected/27603f15-420b-4c05-aaa8-fea5ecb33710-kube-api-access-txvx4\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.141854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-sys\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141807    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27603f15-420b-4c05-aaa8-fea5ecb33710-host\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxmx\" (UniqueName: \"kubernetes.io/projected/3e13d86a-6c05-4f96-a87a-a075c01901ac-kube-api-access-hwxmx\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-sys\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-lib-modules\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27603f15-420b-4c05-aaa8-fea5ecb33710-serviceca\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.141973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh82f\" (UniqueName: \"kubernetes.io/projected/92ad8a17-1941-460c-ba02-ba8d8237f79d-kube-api-access-wh82f\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-systemd\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-hosts-file\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-tmp-dir\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysctl-d\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltx5\" (UniqueName: \"kubernetes.io/projected/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-kube-api-access-4ltx5\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-systemd\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-hosts-file\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:39.142478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142188    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-var-lib-kubelet\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92ad8a17-1941-460c-ba02-ba8d8237f79d-iptables-alerter-script\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-host\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-run\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-sysctl-d\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-system-cni-dir\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-lib-modules\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bx4\" (UniqueName: \"kubernetes.io/projected/589f88ca-b342-488a-9e88-4c60c80dd6bb-kube-api-access-p5bx4\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-run\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-tmp-dir\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-var-lib-kubelet\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e13d86a-6c05-4f96-a87a-a075c01901ac-system-cni-dir\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.143286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-host\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.144098 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.142934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e13d86a-6c05-4f96-a87a-a075c01901ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.144098 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.143014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92ad8a17-1941-460c-ba02-ba8d8237f79d-iptables-alerter-script\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.144098 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.143867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-tmp\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.144854 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.144835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-etc-tuned\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.151790 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.151770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjlz\" (UniqueName: \"kubernetes.io/projected/1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d-kube-api-access-5kjlz\") pod \"tuned-q8bjh\" (UID: \"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d\") " pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.154348 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.154326    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:39.154445 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.154352    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:39.154445 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.154364    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:39.154445 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.154427    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:39.654409771 +0000 UTC m=+4.614295035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:39.155515 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.155490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bx4\" (UniqueName: \"kubernetes.io/projected/589f88ca-b342-488a-9e88-4c60c80dd6bb-kube-api-access-p5bx4\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:39.156005 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.155957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxmx\" (UniqueName: \"kubernetes.io/projected/3e13d86a-6c05-4f96-a87a-a075c01901ac-kube-api-access-hwxmx\") pod \"multus-additional-cni-plugins-kp8xn\" (UID: \"3e13d86a-6c05-4f96-a87a-a075c01901ac\") " pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.156179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.156157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltx5\" (UniqueName: \"kubernetes.io/projected/17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b-kube-api-access-4ltx5\") pod \"node-resolver-7qdqz\" (UID: \"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b\") " pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.156696 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.156679    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvx4\" (UniqueName: \"kubernetes.io/projected/27603f15-420b-4c05-aaa8-fea5ecb33710-kube-api-access-txvx4\") pod \"node-ca-4jfq7\" (UID: \"27603f15-420b-4c05-aaa8-fea5ecb33710\") " pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.157233 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.157214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh82f\" (UniqueName: \"kubernetes.io/projected/92ad8a17-1941-460c-ba02-ba8d8237f79d-kube-api-access-wh82f\") pod \"iptables-alerter-66qpz\" (UID: \"92ad8a17-1941-460c-ba02-ba8d8237f79d\") " pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.230327 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.230297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:34:39.238567 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.238540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:34:39.246993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.246968    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8lsrh"
Sep 17 17:34:39.253818 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.253798    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh"
Sep 17 17:34:39.261474 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.261454    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q8bjh"
Sep 17 17:34:39.270004 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.269984    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66qpz"
Sep 17 17:34:39.277570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.277549    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kp8xn"
Sep 17 17:34:39.290091 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.290070    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7qdqz"
Sep 17 17:34:39.296714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.296696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4jfq7"
Sep 17 17:34:39.302739 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.302720    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:39.545840 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.545752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:39.545974 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.545891    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:39.545974 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.545960    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.545940762 +0000 UTC m=+5.505826016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:39.646403 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.646376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:39.646572 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.646523    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:39.646643 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.646621    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.646603876 +0000 UTC m=+5.606489123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:39.747454 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:39.747415    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:39.747625 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.747560    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:39.747625 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.747584    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:39.747625 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.747595    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:39.747813 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:39.747658    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.747645075 +0000 UTC m=+5.707530338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:40.014376 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.014349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:40.014826 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.014398    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:40.014826 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.014494    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:40.014826 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.014614    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:40.018828 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.018600    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb754174e_ba79_49f2_ae27_2039f91958b4.slice/crio-d4f6f28bacb1e1b6ab5def1f062ad3cdb3fd2ed67e8f02dc067f6b4cda4300ce WatchSource:0}: Error finding container d4f6f28bacb1e1b6ab5def1f062ad3cdb3fd2ed67e8f02dc067f6b4cda4300ce: Status 404 returned error can't find the container with id d4f6f28bacb1e1b6ab5def1f062ad3cdb3fd2ed67e8f02dc067f6b4cda4300ce
Sep 17 17:34:40.020824 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.020481    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:38 +0000 UTC" deadline="2028-05-08 13:23:20.786887378 +0000 UTC"
Sep 17 17:34:40.020824 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.020505    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14371h48m40.76638555s"
Sep 17 17:34:40.020824 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.020505    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9905e7_955d_42f6_9816_a06d3b658ef4.slice/crio-14eb0f32a5c430e4b7dbf3a55a7c60bef375489afe0a09001d830edceb535413 WatchSource:0}: Error finding container 14eb0f32a5c430e4b7dbf3a55a7c60bef375489afe0a09001d830edceb535413: Status 404 returned error can't find the container with id 14eb0f32a5c430e4b7dbf3a55a7c60bef375489afe0a09001d830edceb535413
Sep 17 17:34:40.022531 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.022421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a1efed_a8d2_4761_bc6a_c180cda036d0.slice/crio-fa0e5d57e18f1852aca7b83a934371e52ace70fc050e2bd52213863d2dea2d3e WatchSource:0}: Error finding container fa0e5d57e18f1852aca7b83a934371e52ace70fc050e2bd52213863d2dea2d3e: Status 404 returned error can't find the container with id fa0e5d57e18f1852aca7b83a934371e52ace70fc050e2bd52213863d2dea2d3e
Sep 17 17:34:40.022929 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.022904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"d4f6f28bacb1e1b6ab5def1f062ad3cdb3fd2ed67e8f02dc067f6b4cda4300ce"}
Sep 17 17:34:40.026797 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.026771    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dfa02f_bb9b_49b3_95d8_2a59a1e2f88b.slice/crio-38114e6540e39803b1ac1e6e83256931591f60af8a3fb4584633e5d6ea4329f1 WatchSource:0}: Error finding container 38114e6540e39803b1ac1e6e83256931591f60af8a3fb4584633e5d6ea4329f1: Status 404 returned error can't find the container with id 38114e6540e39803b1ac1e6e83256931591f60af8a3fb4584633e5d6ea4329f1
Sep 17 17:34:40.027176 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.027157    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e13d86a_6c05_4f96_a87a_a075c01901ac.slice/crio-394287c6713d3dcaafb69e00ace035afca43feabdfe0314e7605ff2c4b4b4c8c WatchSource:0}: Error finding container 394287c6713d3dcaafb69e00ace035afca43feabdfe0314e7605ff2c4b4b4c8c: Status 404 returned error can't find the container with id 394287c6713d3dcaafb69e00ace035afca43feabdfe0314e7605ff2c4b4b4c8c
Sep 17 17:34:40.027830 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.027783    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db5de51_17cb_4d7d_94b7_c1cb3f7f4f9d.slice/crio-c30209d2149f4a89316b0d2eb8f1db6181cac0dfdefa7c2e1024da6889221b52 WatchSource:0}: Error finding container c30209d2149f4a89316b0d2eb8f1db6181cac0dfdefa7c2e1024da6889221b52: Status 404 returned error can't find the container with id c30209d2149f4a89316b0d2eb8f1db6181cac0dfdefa7c2e1024da6889221b52
Sep 17 17:34:40.030358 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.030306    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27603f15_420b_4c05_aaa8_fea5ecb33710.slice/crio-13e34b6c73166c1de0c009df6a98d2bbb1e437794ebd87e2b5d21700c07fafbd WatchSource:0}: Error finding container 13e34b6c73166c1de0c009df6a98d2bbb1e437794ebd87e2b5d21700c07fafbd: Status 404 returned error can't find the container with id 13e34b6c73166c1de0c009df6a98d2bbb1e437794ebd87e2b5d21700c07fafbd
Sep 17 17:34:40.033185 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:34:40.032872    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ad8a17_1941_460c_ba02_ba8d8237f79d.slice/crio-13eb13e27e0d4c99b8a3e276974ebacec0402028dd3ff29aca41fbea3daecf0a WatchSource:0}: Error finding container 13eb13e27e0d4c99b8a3e276974ebacec0402028dd3ff29aca41fbea3daecf0a: Status 404 returned error can't find the container with id 13eb13e27e0d4c99b8a3e276974ebacec0402028dd3ff29aca41fbea3daecf0a
Sep 17 17:34:40.555093 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.554999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:40.555261 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.555180    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:40.555261 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.555246    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:34:42.555226615 +0000 UTC m=+7.515111870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:40.656114 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.656075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:40.656295 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.656225    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:40.656295 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.656287    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:34:42.656269106 +0000 UTC m=+7.616154354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:40.757181 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:40.756595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:40.757181 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.756798    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:40.757181 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.756818    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:40.757181 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.756831    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:40.757181 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:40.756888    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:42.756870892 +0000 UTC m=+7.716756143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:41.017216 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.016616    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:41.017216 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:41.016766    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:41.021663 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.021566    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:38 +0000 UTC" deadline="2028-02-27 01:40:02.62413361 +0000 UTC"
Sep 17 17:34:41.021663 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.021613    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12656h5m21.602524173s"
Sep 17 17:34:41.029367 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.029333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8lsrh" event={"ID":"52a1efed-a8d2-4761-bc6a-c180cda036d0","Type":"ContainerStarted","Data":"fa0e5d57e18f1852aca7b83a934371e52ace70fc050e2bd52213863d2dea2d3e"}
Sep 17 17:34:41.032506 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.032443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh" event={"ID":"ee9905e7-955d-42f6-9816-a06d3b658ef4","Type":"ContainerStarted","Data":"14eb0f32a5c430e4b7dbf3a55a7c60bef375489afe0a09001d830edceb535413"}
Sep 17 17:34:41.040924 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.040895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal" event={"ID":"15339a5bbb7d25ed5fa56da3efddf311","Type":"ContainerStarted","Data":"7233c84f47b10d7682237c5813589d56ec34669fbc6158089298a791997abe75"}
Sep 17 17:34:41.049849 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.049169    2561 generic.go:356] "Generic (PLEG): container finished" podID="27263fbfefb9479050311e268f48deef" containerID="dd36575989f3b774ae9df97a6483e89111fc63efac8c68fdbfdfa94edb6afe14" exitCode=0
Sep 17 17:34:41.049849 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.049258    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal" event={"ID":"27263fbfefb9479050311e268f48deef","Type":"ContainerDied","Data":"dd36575989f3b774ae9df97a6483e89111fc63efac8c68fdbfdfa94edb6afe14"}
Sep 17 17:34:41.060953 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.060926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7qdqz" event={"ID":"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b","Type":"ContainerStarted","Data":"38114e6540e39803b1ac1e6e83256931591f60af8a3fb4584633e5d6ea4329f1"}
Sep 17 17:34:41.069813 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.069744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66qpz" event={"ID":"92ad8a17-1941-460c-ba02-ba8d8237f79d","Type":"ContainerStarted","Data":"13eb13e27e0d4c99b8a3e276974ebacec0402028dd3ff29aca41fbea3daecf0a"}
Sep 17 17:34:41.082989 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.082907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v5vhn" event={"ID":"7d1e2c70-905d-42e5-a325-bb57bdb2edcd","Type":"ContainerStarted","Data":"96b9f782a5406e6e1eef17422dd3fb0242d77d4f9a147837d8692d5abb13119c"}
Sep 17 17:34:41.084139 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.083301    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.ec2.internal" podStartSLOduration=3.083283613 podStartE2EDuration="3.083283613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:41.058884337 +0000 UTC m=+6.018769608" watchObservedRunningTime="2026-09-17 17:34:41.083283613 +0000 UTC m=+6.043168886"
Sep 17 17:34:41.089733 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.089673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4jfq7" event={"ID":"27603f15-420b-4c05-aaa8-fea5ecb33710","Type":"ContainerStarted","Data":"13e34b6c73166c1de0c009df6a98d2bbb1e437794ebd87e2b5d21700c07fafbd"}
Sep 17 17:34:41.101066 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.100832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8bjh" event={"ID":"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d","Type":"ContainerStarted","Data":"c30209d2149f4a89316b0d2eb8f1db6181cac0dfdefa7c2e1024da6889221b52"}
Sep 17 17:34:41.110263 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:41.110236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerStarted","Data":"394287c6713d3dcaafb69e00ace035afca43feabdfe0314e7605ff2c4b4b4c8c"}
Sep 17 17:34:42.014994 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:42.014349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:42.014994 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.014449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:42.014994 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:42.014851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:42.014994 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.014947    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:42.142843 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:42.142692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal" event={"ID":"27263fbfefb9479050311e268f48deef","Type":"ContainerStarted","Data":"958166cd8787593941aa1fb2d61fec2ac89b6abd5d63348deafa89d43c3047d8"}
Sep 17 17:34:42.571326 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:42.571280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:42.571468 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.571428    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:42.571555 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.571542    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.571523626 +0000 UTC m=+11.531408879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:42.671802 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:42.671756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:42.671981 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.671933    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:42.672041 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.671994    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.671975787 +0000 UTC m=+11.631861041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:42.773383 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:42.773341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:42.786664 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.786128    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:42.786664 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.786160    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:42.786664 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.786174    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:42.786664 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:42.786241    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:46.786223019 +0000 UTC m=+11.746108273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:43.016123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:43.016040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:43.016281 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:43.016171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:44.014773 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:44.014251    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:44.014773 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:44.014399    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:44.014773 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:44.014658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:44.015346 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:44.014779    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:45.014820 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:45.014310    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:45.015236 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:45.014939    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:46.014374 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:46.014340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:46.014566 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:46.014340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:46.014566 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.014477    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:46.014566 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.014519    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:46.606904 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:46.606863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:46.607362 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.607078    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:46.607362 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.607143    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.607123772 +0000 UTC m=+19.567009034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:46.707715 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:46.707681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:46.707901 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.707828    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:46.707901 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.707892    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.707872965 +0000 UTC m=+19.667758227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:46.808297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:46.808257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:46.808479 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.808453    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:46.808479 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.808471    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:46.808591 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.808483    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:46.808591 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:46.808540    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.808523494 +0000 UTC m=+19.768408748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:47.015758 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:47.015640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:47.015758 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:47.015747    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:48.015621 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:48.015100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:48.015621 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:48.015142    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:48.015621 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:48.015223    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:48.015621 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:48.015359    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:49.015077 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:49.015040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:49.015243 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:49.015173    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:50.014091 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:50.014050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:50.014091 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:50.014095    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:50.014577 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:50.014188    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:50.014577 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:50.014330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:51.014670 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:51.014624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:51.015219 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:51.014755    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:52.015070 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:52.015035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:52.015513 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:52.015162    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:52.015513 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:52.015228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:52.015513 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:52.015341    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:53.015157 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:53.015127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:53.015548 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:53.015257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:54.014115 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:54.014081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:54.014297 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.014211    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:54.014297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:54.014254    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:54.014408 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.014336    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:54.657887 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:54.657854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:54.658287 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.658020    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:54.658287 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.658080    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:35:10.658064539 +0000 UTC m=+35.617949786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:54.758564 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:54.758524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:54.758749 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.758709    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:54.758813 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.758777    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:35:10.758759571 +0000 UTC m=+35.718644831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:54.859783 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:54.859744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:54.859932 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.859900    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:54.859932 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.859919    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:54.859932 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.859930    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:54.860056 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:54.859976    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:10.859962358 +0000 UTC m=+35.819847607 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:55.014548 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:55.014447    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:55.014720 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:55.014587    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:56.014712 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:56.014673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:56.014712 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:56.014702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:56.015122 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:56.014806    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:56.015122 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:56.014940    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:57.014534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:57.014504    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:57.014697 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:57.014577    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:58.015327 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.014999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:34:58.016124 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.015025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:34:58.016124 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:58.015449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:34:58.016124 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:58.015500    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:34:58.175649 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.175594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q8bjh" event={"ID":"1db5de51-17cb-4d7d-94b7-c1cb3f7f4f9d","Type":"ContainerStarted","Data":"77c76b67325004799eb1c4e2b2705625ef37529e66054c193e917aa6034be77b"}
Sep 17 17:34:58.177414 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.177386    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e13d86a-6c05-4f96-a87a-a075c01901ac" containerID="c03c3a45e0c3bebd5d0dcb4031da65e4516ce8b8ea3973c93cf6f416ed8281c6" exitCode=0
Sep 17 17:34:58.177535 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.177472    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerDied","Data":"c03c3a45e0c3bebd5d0dcb4031da65e4516ce8b8ea3973c93cf6f416ed8281c6"}
Sep 17 17:34:58.179394 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.179367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8lsrh" event={"ID":"52a1efed-a8d2-4761-bc6a-c180cda036d0","Type":"ContainerStarted","Data":"a95191d15f8ef267e0dfe681c14517141ec0edf9ff2cbcc47cc605abe695f9a5"}
Sep 17 17:34:58.181250 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.181229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh" event={"ID":"ee9905e7-955d-42f6-9816-a06d3b658ef4","Type":"ContainerStarted","Data":"cd950a26f99f9b93f1f5fd8723225e24f63f449daa0c2615423f9f639d835944"}
Sep 17 17:34:58.184617 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.184595    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"d92de23d59579c26a6c688bded15fc3289adfbe2a3ad491cbc1768efd7f011d4"}
Sep 17 17:34:58.184715 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.184682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"e48c1f88fcf9aac7e1dadbb45b18ba5af8f18503d7073e9cd1cfdd582ac135b1"}
Sep 17 17:34:58.184772 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.184721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"3b8723223293e27748d422afd64f8951066aa0128ea413266a4460b97ec53cdb"}
Sep 17 17:34:58.184772 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.184761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"657c9d043f05771ccb81f590c59061ec5a754f94edc0bf7cd965a41c0920d5fc"}
Sep 17 17:34:58.184875 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.184796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"506b3baf86e9f44d28fe7230e7ab4abd843301bf2fe829f5f271745a573d3e38"}
Sep 17 17:34:58.184875 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.184842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"3ee8727c146e3be76bd711433c5cd7fe139fd8d90ca320f0686c6a3849077a4d"}
Sep 17 17:34:58.186263 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.186241    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7qdqz" event={"ID":"17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b","Type":"ContainerStarted","Data":"d127ec251a17c517675b2c7da99c4347f3272262cc4f1355598399642c4fc2c1"}
Sep 17 17:34:58.187807 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.187776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v5vhn" event={"ID":"7d1e2c70-905d-42e5-a325-bb57bdb2edcd","Type":"ContainerStarted","Data":"dd4d4a0018c7e8a71bf1e98dd0a2945c6a7a5add892c958c47a17dd14f35e08d"}
Sep 17 17:34:58.189219 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.189197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4jfq7" event={"ID":"27603f15-420b-4c05-aaa8-fea5ecb33710","Type":"ContainerStarted","Data":"640947e8cd69936ce48ea092e332ea448e9b13b88eb949d94f5f0d8c689c4289"}
Sep 17 17:34:58.191621 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.191579    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q8bjh" podStartSLOduration=3.806803939 podStartE2EDuration="21.191562702s" totalImagesPullingTime="17.384758763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.191175226 +0000 UTC m=+23.151060498" watchObservedRunningTime="2026-09-17 17:34:58.191562702 +0000 UTC m=+23.151447973"
Sep 17 17:34:58.192003 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.191972    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.ec2.internal" podStartSLOduration=20.191966828 podStartE2EDuration="20.191966828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:42.157531997 +0000 UTC m=+7.117417268" watchObservedRunningTime="2026-09-17 17:34:58.191966828 +0000 UTC m=+23.151852099"
Sep 17 17:34:58.228558 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.228511    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7qdqz" podStartSLOduration=3.843400218 podStartE2EDuration="21.228497003s" totalImagesPullingTime="17.385096785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.228191393 +0000 UTC m=+23.188076664" watchObservedRunningTime="2026-09-17 17:34:58.228497003 +0000 UTC m=+23.188382277"
Sep 17 17:34:58.269142 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.269094    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4jfq7" podStartSLOduration=3.889387356 podStartE2EDuration="21.269076429s" totalImagesPullingTime="17.379689073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.248580294 +0000 UTC m=+23.208465568" watchObservedRunningTime="2026-09-17 17:34:58.269076429 +0000 UTC m=+23.228961700"
Sep 17 17:34:58.269439 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.269398    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v5vhn" podStartSLOduration=11.431233588 podStartE2EDuration="21.269389302s" totalImagesPullingTime="9.838155714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.268761615 +0000 UTC m=+23.228646886" watchObservedRunningTime="2026-09-17 17:34:58.269389302 +0000 UTC m=+23.229274646"
Sep 17 17:34:58.287062 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.287027    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8lsrh" podStartSLOduration=3.845111603 podStartE2EDuration="21.287016036s" totalImagesPullingTime="17.441904433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.285830979 +0000 UTC m=+23.245716263" watchObservedRunningTime="2026-09-17 17:34:58.287016036 +0000 UTC m=+23.246901306"
Sep 17 17:34:58.422538 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.422514    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:58.999113 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:58.999023    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:58.422533302Z","UUID":"3fdeb74d-3741-4747-806f-482fd52690d6","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:59.000555 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:59.000530    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:59.000555 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:59.000560    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:59.014548 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:59.014521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:34:59.014698 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:34:59.014676    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:34:59.194865 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:59.194780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh" event={"ID":"ee9905e7-955d-42f6-9816-a06d3b658ef4","Type":"ContainerStarted","Data":"68a3525102663d84e496db7a1ed8fbd1df5618fd7b6302809e23d515524efd7f"}
Sep 17 17:34:59.196723 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:34:59.196689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66qpz" event={"ID":"92ad8a17-1941-460c-ba02-ba8d8237f79d","Type":"ContainerStarted","Data":"da88efc98633efa034aaa3144a7ddcccf0a68ec1932558c727d6b9b617a73f54"}
Sep 17 17:35:00.015204 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.014987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:00.015348 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.015089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:00.015348 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:00.015301    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:00.015453 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:00.015393    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:00.201762 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.201671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh" event={"ID":"ee9905e7-955d-42f6-9816-a06d3b658ef4","Type":"ContainerStarted","Data":"78444c0ca80749826767688c6635b897e92ef518c25057c87d60d6ad0fa2adbf"}
Sep 17 17:35:00.205579 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.205510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"cc611145716797bd09f53cde639f0a76b748a7798669fd3df9cbde34a6fd9204"}
Sep 17 17:35:00.238071 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.238015    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-66qpz" podStartSLOduration=5.864886863 podStartE2EDuration="23.237998801s" totalImagesPullingTime="17.373111938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:59.212690855 +0000 UTC m=+24.172576126" watchObservedRunningTime="2026-09-17 17:35:00.237998801 +0000 UTC m=+25.197884070"
Sep 17 17:35:00.659463 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.659375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:35:00.660018 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.659998    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:35:00.679310 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:00.679262    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-828fh" podStartSLOduration=4.701835145 podStartE2EDuration="23.679245034s" totalImagesPullingTime="18.977409889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:00.238428865 +0000 UTC m=+25.198314135" watchObservedRunningTime="2026-09-17 17:35:00.679245034 +0000 UTC m=+25.639130285"
Sep 17 17:35:01.014907 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:01.014842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:01.015048 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:01.014965    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:35:01.207682 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:01.207653    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:35:01.208260 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:01.208241    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v5vhn"
Sep 17 17:35:02.014763 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:02.014738    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:02.014931 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:02.014745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:02.014931 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:02.014830    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:02.014931 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:02.014925    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:02.217650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:02.217513    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" event={"ID":"b754174e-ba79-49f2-ae27-2039f91958b4","Type":"ContainerStarted","Data":"6e7e8c6b118f51412049e0c83782d225c205daac959100376f4db3aac9ae769e"}
Sep 17 17:35:02.253855 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:02.253799    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm" podStartSLOduration=7.836965106 podStartE2EDuration="25.25377934s" totalImagesPullingTime="17.416814234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:02.253674776 +0000 UTC m=+27.213560046" watchObservedRunningTime="2026-09-17 17:35:02.25377934 +0000 UTC m=+27.213664612"
Sep 17 17:35:03.014297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.014134    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:03.014434 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:03.014365    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:35:03.221581 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.221549    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e13d86a-6c05-4f96-a87a-a075c01901ac" containerID="55b1037fe780f7befe8beaae44115257c0d9116795462c7178eeabecafad3835" exitCode=0
Sep 17 17:35:03.222045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.221653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerDied","Data":"55b1037fe780f7befe8beaae44115257c0d9116795462c7178eeabecafad3835"}
Sep 17 17:35:03.222281 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.222239    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:35:03.223013 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.222323    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:35:03.223013 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.222375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:35:03.236965 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.236945    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:35:03.237060 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:03.237047    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:35:04.014907 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.014879    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:04.014907 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.014902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:04.015126 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:04.014995    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:04.015126 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:04.015102    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:04.258499 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.258211    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qdh2w"]
Sep 17 17:35:04.258499 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.258372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:04.258499 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:04.258486    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:04.259020 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.258758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v9xw9"]
Sep 17 17:35:04.259020 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.258847    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:04.259020 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:04.258925    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:35:04.259661 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.259617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6dz7"]
Sep 17 17:35:04.259775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:04.259718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:04.259838 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:04.259808    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:05.228457 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:05.228426    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e13d86a-6c05-4f96-a87a-a075c01901ac" containerID="83ca91e1aa426f7b1488331b4bca0c5745f8bef79bd90260df95969a44969921" exitCode=0
Sep 17 17:35:05.228690 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:05.228519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerDied","Data":"83ca91e1aa426f7b1488331b4bca0c5745f8bef79bd90260df95969a44969921"}
Sep 17 17:35:06.014590 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:06.014561    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:06.014590 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:06.014590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:06.014982 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:06.014688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:06.014982 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:06.014694    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:06.014982 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:06.014748    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:35:06.014982 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:06.014817    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:07.235852 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:07.235618    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e13d86a-6c05-4f96-a87a-a075c01901ac" containerID="668ac6fc7d6641e22ed9fcf293f891b660b1b93c6c0868bb337743ce5c9ac63a" exitCode=0
Sep 17 17:35:07.236236 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:07.235664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerDied","Data":"668ac6fc7d6641e22ed9fcf293f891b660b1b93c6c0868bb337743ce5c9ac63a"}
Sep 17 17:35:08.014584 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:08.014555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:08.014584 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:08.014574    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:08.014850 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:08.014589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:08.014850 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:08.014683    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:35:08.014850 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:08.014775    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:08.014948 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:08.014860    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:10.014779 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.014742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:10.015385 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.014787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:10.015385 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.014852    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:10.015385 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.014873    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v9xw9" podUID="2f88a7bc-7803-4d65-bf80-1a884f0f3550"
Sep 17 17:35:10.015385 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.014929    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qdh2w" podUID="589f88ca-b342-488a-9e88-4c60c80dd6bb"
Sep 17 17:35:10.015385 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.015005    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d6dz7" podUID="d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e"
Sep 17 17:35:10.062463 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.062434    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-122.ec2.internal" event="NodeReady"
Sep 17 17:35:10.062654 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.062573    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:35:10.112196 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.112165    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cbfd45bdf-c6frm"]
Sep 17 17:35:10.128370 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.128343    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.133966 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.133933    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:35:10.134285 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.134266    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-55z8g\""
Sep 17 17:35:10.134740 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.134721    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:35:10.134984 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.134958    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:35:10.140045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.140016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:35:10.143419 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.143395    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bcllq"]
Sep 17 17:35:10.161567 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.161549    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.161694 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.161546    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vssft"]
Sep 17 17:35:10.167935 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.167518    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wr4qk\""
Sep 17 17:35:10.167935 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.167774    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:35:10.167935 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.167778    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:35:10.169123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.169106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:35:10.173690 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.173645    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbfd45bdf-c6frm"]
Sep 17 17:35:10.173690 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.173681    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vssft"]
Sep 17 17:35:10.173810 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.173698    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bcllq"]
Sep 17 17:35:10.173810 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.173765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.178511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.178489    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:35:10.178617 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.178543    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dng7x\""
Sep 17 17:35:10.178617 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.178591    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:35:10.264214 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-trusted-ca\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-bound-sa-token\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4n6s\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-kube-api-access-r4n6s\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9faa45-7dfc-4569-9c1f-30a623d562e7-config-volume\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.264378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-installation-pull-secrets\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.264551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lp74\" (UniqueName: \"kubernetes.io/projected/cc9faa45-7dfc-4569-9c1f-30a623d562e7-kube-api-access-9lp74\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.264551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-ca-trust-extracted\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-image-registry-private-configuration\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264491    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94e2c20-8bbd-47d1-b3d1-391b109eccae-cert\") pod \"ingress-canary-bcllq\" (UID: \"a94e2c20-8bbd-47d1-b3d1-391b109eccae\") " pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.264792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9faa45-7dfc-4569-9c1f-30a623d562e7-tmp-dir\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.264792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.264792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9phm\" (UniqueName: \"kubernetes.io/projected/a94e2c20-8bbd-47d1-b3d1-391b109eccae-kube-api-access-r9phm\") pod \"ingress-canary-bcllq\" (UID: \"a94e2c20-8bbd-47d1-b3d1-391b109eccae\") " pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.264792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.264614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-certificates\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365099 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9faa45-7dfc-4569-9c1f-30a623d562e7-tmp-dir\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9phm\" (UniqueName: \"kubernetes.io/projected/a94e2c20-8bbd-47d1-b3d1-391b109eccae-kube-api-access-r9phm\") pod \"ingress-canary-bcllq\" (UID: \"a94e2c20-8bbd-47d1-b3d1-391b109eccae\") " pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-certificates\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-trusted-ca\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-bound-sa-token\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.365254    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:10.365286 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.365269    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:10.365609 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.365328    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:10.865309363 +0000 UTC m=+35.825194630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:10.365609 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4n6s\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-kube-api-access-r4n6s\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365609 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9faa45-7dfc-4569-9c1f-30a623d562e7-config-volume\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.365609 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9faa45-7dfc-4569-9c1f-30a623d562e7-tmp-dir\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.365850 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-installation-pull-secrets\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365850 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.365850 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lp74\" (UniqueName: \"kubernetes.io/projected/cc9faa45-7dfc-4569-9c1f-30a623d562e7-kube-api-access-9lp74\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.365850 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-ca-trust-extracted\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.365850 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-image-registry-private-configuration\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.366080 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94e2c20-8bbd-47d1-b3d1-391b109eccae-cert\") pod \"ingress-canary-bcllq\" (UID: \"a94e2c20-8bbd-47d1-b3d1-391b109eccae\") " pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.366080 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.365974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-certificates\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.366166 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.366092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9faa45-7dfc-4569-9c1f-30a623d562e7-config-volume\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.366374 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.366291    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:10.366374 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.366351    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:10.866334053 +0000 UTC m=+35.826219314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:10.366509 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.366490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-ca-trust-extracted\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.366917 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.366619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-trusted-ca\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.370239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.370218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-image-registry-private-configuration\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.370331 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.370278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94e2c20-8bbd-47d1-b3d1-391b109eccae-cert\") pod \"ingress-canary-bcllq\" (UID: \"a94e2c20-8bbd-47d1-b3d1-391b109eccae\") " pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.370380 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.370341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-installation-pull-secrets\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.390009 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.389982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lp74\" (UniqueName: \"kubernetes.io/projected/cc9faa45-7dfc-4569-9c1f-30a623d562e7-kube-api-access-9lp74\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.390319 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.390277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4n6s\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-kube-api-access-r4n6s\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.390831 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.390812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9phm\" (UniqueName: \"kubernetes.io/projected/a94e2c20-8bbd-47d1-b3d1-391b109eccae-kube-api-access-r9phm\") pod \"ingress-canary-bcllq\" (UID: \"a94e2c20-8bbd-47d1-b3d1-391b109eccae\") " pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.391825 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.391804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-bound-sa-token\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.472842 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.472809    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bcllq"
Sep 17 17:35:10.669365 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.669289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:10.669527 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.669419    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:10.669527 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.669488    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret podName:d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e nodeName:}" failed. No retries permitted until 2026-09-17 17:35:42.669471261 +0000 UTC m=+67.629356522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret") pod "global-pull-secret-syncer-d6dz7" (UID: "d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:10.770040 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.770008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:10.770215 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.770137    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:10.770260 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.770223    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:35:42.770201911 +0000 UTC m=+67.730087185 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:10.871088 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.871054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:10.871271 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.871098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:10.871271 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:10.871162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:10.871271 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871220    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:10.871271 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871262    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871276    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871279    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871298    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871308    2561 projected.go:196] Error preparing data for projected volume kube-api-access-s5xtw for pod openshift-network-diagnostics/network-check-target-v9xw9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871295    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.871274731 +0000 UTC m=+36.831159982 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871358    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.871340601 +0000 UTC m=+36.831225852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:10.871452 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:10.871372    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw podName:2f88a7bc-7803-4d65-bf80-1a884f0f3550 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:42.871363418 +0000 UTC m=+67.831248666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5xtw" (UniqueName: "kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw") pod "network-check-target-v9xw9" (UID: "2f88a7bc-7803-4d65-bf80-1a884f0f3550") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:11.633062 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:11.633035    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bcllq"]
Sep 17 17:35:11.639748 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:35:11.639724    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94e2c20_8bbd_47d1_b3d1_391b109eccae.slice/crio-47a9b857a67f502a6cab27bccff663316e146ee2d91c84faf512864bef64ed9c WatchSource:0}: Error finding container 47a9b857a67f502a6cab27bccff663316e146ee2d91c84faf512864bef64ed9c: Status 404 returned error can't find the container with id 47a9b857a67f502a6cab27bccff663316e146ee2d91c84faf512864bef64ed9c
Sep 17 17:35:11.879619 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:11.879567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:11.879792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:11.879623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:11.879792 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:11.879725    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:11.879792 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:11.879736    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:11.879792 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:11.879779    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:13.879767506 +0000 UTC m=+38.839652754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:11.879923 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:11.879723    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:11.879923 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:11.879845    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:13.879830845 +0000 UTC m=+38.839716115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:12.014884 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.014858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:12.015018 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.014946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:12.015124 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.015104    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:12.022593 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.022574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:12.023549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.023533    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j84n5\""
Sep 17 17:35:12.023549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.023534    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:12.023746 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.023613    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:12.023746 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.023682    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:12.023908 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.023894    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vwwqh\""
Sep 17 17:35:12.252525 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.252492    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e13d86a-6c05-4f96-a87a-a075c01901ac" containerID="b639a47b8a411646eb4d69ecb0021e0ebf505b516d4311f60e7d631d3189e482" exitCode=0
Sep 17 17:35:12.252753 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.252530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerDied","Data":"b639a47b8a411646eb4d69ecb0021e0ebf505b516d4311f60e7d631d3189e482"}
Sep 17 17:35:12.254006 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:12.253973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bcllq" event={"ID":"a94e2c20-8bbd-47d1-b3d1-391b109eccae","Type":"ContainerStarted","Data":"47a9b857a67f502a6cab27bccff663316e146ee2d91c84faf512864bef64ed9c"}
Sep 17 17:35:13.259864 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:13.259829    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e13d86a-6c05-4f96-a87a-a075c01901ac" containerID="514d355727a2268ecfc55868417845545c5717eca325859c7879e3ab67ffe734" exitCode=0
Sep 17 17:35:13.260466 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:13.259880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerDied","Data":"514d355727a2268ecfc55868417845545c5717eca325859c7879e3ab67ffe734"}
Sep 17 17:35:13.898656 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:13.898620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:13.898740 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:13.898725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:13.898780 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:13.898752    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:13.898817 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:13.898799    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.898784936 +0000 UTC m=+42.858670184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:13.898868 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:13.898856    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:13.898906 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:13.898870    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:13.898937 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:13.898922    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:17.898906457 +0000 UTC m=+42.858791705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:14.267471 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:14.267260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" event={"ID":"3e13d86a-6c05-4f96-a87a-a075c01901ac","Type":"ContainerStarted","Data":"cf342f0806d2894423fab28c6a65dac845bf16101ab59dcd98b1367e712c99c2"}
Sep 17 17:35:14.269461 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:14.269435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bcllq" event={"ID":"a94e2c20-8bbd-47d1-b3d1-391b109eccae","Type":"ContainerStarted","Data":"8824be80ce86519d77a1855bdbfedeca062928b4667e549cd0718a9966bde44b"}
Sep 17 17:35:14.299378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:14.299328    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kp8xn" podStartSLOduration=9.279495517 podStartE2EDuration="37.299313293s" totalImagesPullingTime="28.019817776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.298809696 +0000 UTC m=+39.258694965" watchObservedRunningTime="2026-09-17 17:35:14.299313293 +0000 UTC m=+39.259198563"
Sep 17 17:35:14.320327 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:14.320282    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bcllq" podStartSLOduration=1.8328347520000001 podStartE2EDuration="4.320265988s" totalImagesPullingTime="2.487431236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.319736648 +0000 UTC m=+39.279621918" watchObservedRunningTime="2026-09-17 17:35:14.320265988 +0000 UTC m=+39.280151257"
Sep 17 17:35:17.929279 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:17.929233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:17.929684 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:17.929306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:17.929684 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:17.929393    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:17.929684 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:17.929411    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:17.929684 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:17.929395    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:17.929684 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:17.929470    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:25.929452458 +0000 UTC m=+50.889337727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:17.929684 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:17.929490    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:25.929482003 +0000 UTC m=+50.889367264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:25.982533 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:25.982490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:25.982973 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:25.982578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:25.982973 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:25.982662    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:25.982973 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:25.982714    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:25.982973 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:25.982731    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:25.982973 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:25.982732    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:41.982716616 +0000 UTC m=+66.942601891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:25.982973 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:25.982792    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:41.982779231 +0000 UTC m=+66.942664479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:35.245560 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:35.245530    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pddbm"
Sep 17 17:35:41.992081 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:41.992045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:35:41.992451 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:41.992103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:35:41.992451 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:41.992184    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:41.992451 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:41.992203    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:35:41.992451 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:41.992188    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:41.992451 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:41.992271    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.992250824 +0000 UTC m=+98.952136072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:35:41.992451 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:41.992289    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:13.992282215 +0000 UTC m=+98.952167463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:42.697342 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.697311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:42.700852 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.700834    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:42.711762 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.711735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e-original-pull-secret\") pod \"global-pull-secret-syncer-d6dz7\" (UID: \"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e\") " pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:42.798001 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.797973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:35:42.801250 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.801230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:42.809017 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:42.809002    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:42.809098 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:35:42.809063    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs podName:589f88ca-b342-488a-9e88-4c60c80dd6bb nodeName:}" failed. No retries permitted until 2026-09-17 17:36:46.809044612 +0000 UTC m=+131.768929880 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs") pod "network-metrics-daemon-qdh2w" (UID: "589f88ca-b342-488a-9e88-4c60c80dd6bb") : secret "metrics-daemon-secret" not found
Sep 17 17:35:42.898844 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.898813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:42.901974 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.901944    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:42.912899 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.912880    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:42.922754 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.922730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xtw\" (UniqueName: \"kubernetes.io/projected/2f88a7bc-7803-4d65-bf80-1a884f0f3550-kube-api-access-s5xtw\") pod \"network-check-target-v9xw9\" (UID: \"2f88a7bc-7803-4d65-bf80-1a884f0f3550\") " pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:42.927575 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.927559    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vwwqh\""
Sep 17 17:35:42.929590 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.929578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d6dz7"
Sep 17 17:35:42.934824 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:42.934808    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:43.218430 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:43.218389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d6dz7"]
Sep 17 17:35:43.221317 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:35:43.221290    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c4029f_d3c5_4c0e_82fc_0f9f48968e8e.slice/crio-73a36f8bda68b1fc7886373caaebd06525b48a373e9eef50c7ea421a9b3e6f2f WatchSource:0}: Error finding container 73a36f8bda68b1fc7886373caaebd06525b48a373e9eef50c7ea421a9b3e6f2f: Status 404 returned error can't find the container with id 73a36f8bda68b1fc7886373caaebd06525b48a373e9eef50c7ea421a9b3e6f2f
Sep 17 17:35:43.222261 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:43.222241    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v9xw9"]
Sep 17 17:35:43.225616 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:35:43.225593    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f88a7bc_7803_4d65_bf80_1a884f0f3550.slice/crio-20c8d6fa8879e607a271be7febdb7b4f751e67dc9db2aa674a5e97e8dc8251d9 WatchSource:0}: Error finding container 20c8d6fa8879e607a271be7febdb7b4f751e67dc9db2aa674a5e97e8dc8251d9: Status 404 returned error can't find the container with id 20c8d6fa8879e607a271be7febdb7b4f751e67dc9db2aa674a5e97e8dc8251d9
Sep 17 17:35:43.338700 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:43.338671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v9xw9" event={"ID":"2f88a7bc-7803-4d65-bf80-1a884f0f3550","Type":"ContainerStarted","Data":"20c8d6fa8879e607a271be7febdb7b4f751e67dc9db2aa674a5e97e8dc8251d9"}
Sep 17 17:35:43.339765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:43.339743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6dz7" event={"ID":"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e","Type":"ContainerStarted","Data":"73a36f8bda68b1fc7886373caaebd06525b48a373e9eef50c7ea421a9b3e6f2f"}
Sep 17 17:35:48.355257 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:48.355217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d6dz7" event={"ID":"d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e","Type":"ContainerStarted","Data":"286b8f968d4d192dc7bec29e70f2d2d6be6848593969302b2d254243e795866e"}
Sep 17 17:35:48.356767 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:48.356746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v9xw9" event={"ID":"2f88a7bc-7803-4d65-bf80-1a884f0f3550","Type":"ContainerStarted","Data":"785402f66f8de0a3a3e6ae05835939898ba539805faeb53c7a23d726fc390e2c"}
Sep 17 17:35:48.356852 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:48.356827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:35:48.374012 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:48.373969    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d6dz7" podStartSLOduration=67.117691351 podStartE2EDuration="1m11.373956696s" totalImagesPullingTime="4.256265345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:48.373015482 +0000 UTC m=+73.332900750" watchObservedRunningTime="2026-09-17 17:35:48.373956696 +0000 UTC m=+73.333841966"
Sep 17 17:35:48.389925 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:35:48.389889    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v9xw9" podStartSLOduration=67.14136055 podStartE2EDuration="1m11.389874914s" totalImagesPullingTime="4.248514364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:48.388760997 +0000 UTC m=+73.348646268" watchObservedRunningTime="2026-09-17 17:35:48.389874914 +0000 UTC m=+73.349760183"
Sep 17 17:36:14.010324 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:14.010286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:36:14.010779 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:14.010344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:36:14.010779 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:14.010424    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:14.010779 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:14.010424    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:14.010779 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:14.010485    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls podName:cc9faa45-7dfc-4569-9c1f-30a623d562e7 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:18.010471172 +0000 UTC m=+162.970356420 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls") pod "dns-default-vssft" (UID: "cc9faa45-7dfc-4569-9c1f-30a623d562e7") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:14.010779 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:14.010434    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbfd45bdf-c6frm: secret "image-registry-tls" not found
Sep 17 17:36:14.010779 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:14.010550    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls podName:f19b9e24-40a5-4a33-8f8e-24a299fb0f02 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:18.010538692 +0000 UTC m=+162.970423956 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls") pod "image-registry-cbfd45bdf-c6frm" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02") : secret "image-registry-tls" not found
Sep 17 17:36:19.361957 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:19.361924    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v9xw9"
Sep 17 17:36:24.060477 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.060447    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"]
Sep 17 17:36:24.064440 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.064415    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-mjfcw"]
Sep 17 17:36:24.064590 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.064552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.067153 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.067132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.067270 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.067163    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:36:24.067270 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.067175    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:24.068201 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.068181    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:24.068316 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.068212    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7cr72\""
Sep 17 17:36:24.071392 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.071364    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cv7kt\""
Sep 17 17:36:24.071488 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.071435    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:24.071545 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.071440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:24.072475 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.072459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:36:24.072911 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.072873    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:36:24.076084 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.076066    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:36:24.077846 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.077829    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"]
Sep 17 17:36:24.087655 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.087622    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-mjfcw"]
Sep 17 17:36:24.178320 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.178292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2db503d-1499-4ea7-a841-ab983806a9f6-serving-cert\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.178320 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.178319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db503d-1499-4ea7-a841-ab983806a9f6-config\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.178526 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.178358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2db503d-1499-4ea7-a841-ab983806a9f6-trusted-ca\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.178526 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.178392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2dgm\" (UniqueName: \"kubernetes.io/projected/f2db503d-1499-4ea7-a841-ab983806a9f6-kube-api-access-w2dgm\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.178526 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.178409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q5kw\" (UniqueName: \"kubernetes.io/projected/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-kube-api-access-4q5kw\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.178526 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.178431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.279669 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.279611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2db503d-1499-4ea7-a841-ab983806a9f6-serving-cert\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.279752 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.279682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db503d-1499-4ea7-a841-ab983806a9f6-config\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.279752 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.279745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2db503d-1499-4ea7-a841-ab983806a9f6-trusted-ca\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.279833 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.279785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2dgm\" (UniqueName: \"kubernetes.io/projected/f2db503d-1499-4ea7-a841-ab983806a9f6-kube-api-access-w2dgm\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.279833 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.279813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q5kw\" (UniqueName: \"kubernetes.io/projected/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-kube-api-access-4q5kw\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.279944 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.279844    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.280005 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:24.279945    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:24.280084 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:24.280069    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls podName:a413e23b-1cf0-43dc-ada7-1f3b06830c5f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.780049685 +0000 UTC m=+109.739934935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-49pdh" (UID: "a413e23b-1cf0-43dc-ada7-1f3b06830c5f") : secret "samples-operator-tls" not found
Sep 17 17:36:24.280421 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.280400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db503d-1499-4ea7-a841-ab983806a9f6-config\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.280518 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.280503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2db503d-1499-4ea7-a841-ab983806a9f6-trusted-ca\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.281970 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.281940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2db503d-1499-4ea7-a841-ab983806a9f6-serving-cert\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.301190 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.301166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2dgm\" (UniqueName: \"kubernetes.io/projected/f2db503d-1499-4ea7-a841-ab983806a9f6-kube-api-access-w2dgm\") pod \"console-operator-fd4bb46c-mjfcw\" (UID: \"f2db503d-1499-4ea7-a841-ab983806a9f6\") " pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.302869 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.302852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q5kw\" (UniqueName: \"kubernetes.io/projected/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-kube-api-access-4q5kw\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.380290 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.380194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:24.650073 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.649988    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-mjfcw"]
Sep 17 17:36:24.654575 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:36:24.654551    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2db503d_1499_4ea7_a841_ab983806a9f6.slice/crio-110dc8f9196a359df5b2f98e8ab8e6f47c2173aea070e5b1fdb7421edb692411 WatchSource:0}: Error finding container 110dc8f9196a359df5b2f98e8ab8e6f47c2173aea070e5b1fdb7421edb692411: Status 404 returned error can't find the container with id 110dc8f9196a359df5b2f98e8ab8e6f47c2173aea070e5b1fdb7421edb692411
Sep 17 17:36:24.784086 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:24.784054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:24.784234 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:24.784151    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:24.784234 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:24.784214    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls podName:a413e23b-1cf0-43dc-ada7-1f3b06830c5f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:25.784199586 +0000 UTC m=+110.744084834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-49pdh" (UID: "a413e23b-1cf0-43dc-ada7-1f3b06830c5f") : secret "samples-operator-tls" not found
Sep 17 17:36:25.447032 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:25.446998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" event={"ID":"f2db503d-1499-4ea7-a841-ab983806a9f6","Type":"ContainerStarted","Data":"110dc8f9196a359df5b2f98e8ab8e6f47c2173aea070e5b1fdb7421edb692411"}
Sep 17 17:36:25.790797 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:25.790752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:25.790986 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:25.790962    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:25.791073 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:25.791061    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls podName:a413e23b-1cf0-43dc-ada7-1f3b06830c5f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:27.791029686 +0000 UTC m=+112.750914951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-49pdh" (UID: "a413e23b-1cf0-43dc-ada7-1f3b06830c5f") : secret "samples-operator-tls" not found
Sep 17 17:36:26.451204 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:26.451178    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/0.log"
Sep 17 17:36:26.451576 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:26.451214    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2db503d-1499-4ea7-a841-ab983806a9f6" containerID="0ec3040953db64da4560c962c0f75c543e4cf0dfb76907febbb3fcebaf08c17f" exitCode=255
Sep 17 17:36:26.451576 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:26.451239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" event={"ID":"f2db503d-1499-4ea7-a841-ab983806a9f6","Type":"ContainerDied","Data":"0ec3040953db64da4560c962c0f75c543e4cf0dfb76907febbb3fcebaf08c17f"}
Sep 17 17:36:26.451576 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:26.451447    2561 scope.go:117] "RemoveContainer" containerID="0ec3040953db64da4560c962c0f75c543e4cf0dfb76907febbb3fcebaf08c17f"
Sep 17 17:36:27.454571 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.454544    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/1.log"
Sep 17 17:36:27.454980 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.454910    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/0.log"
Sep 17 17:36:27.454980 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.454941    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2db503d-1499-4ea7-a841-ab983806a9f6" containerID="697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f" exitCode=255
Sep 17 17:36:27.454980 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.454972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" event={"ID":"f2db503d-1499-4ea7-a841-ab983806a9f6","Type":"ContainerDied","Data":"697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f"}
Sep 17 17:36:27.455081 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.455021    2561 scope.go:117] "RemoveContainer" containerID="0ec3040953db64da4560c962c0f75c543e4cf0dfb76907febbb3fcebaf08c17f"
Sep 17 17:36:27.455257 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.455241    2561 scope.go:117] "RemoveContainer" containerID="697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f"
Sep 17 17:36:27.455394 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:27.455372    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-mjfcw_openshift-console-operator(f2db503d-1499-4ea7-a841-ab983806a9f6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podUID="f2db503d-1499-4ea7-a841-ab983806a9f6"
Sep 17 17:36:27.803213 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.803169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:27.803388 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:27.803300    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:27.803388 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:27.803354    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls podName:a413e23b-1cf0-43dc-ada7-1f3b06830c5f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.80333999 +0000 UTC m=+116.763225239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-49pdh" (UID: "a413e23b-1cf0-43dc-ada7-1f3b06830c5f") : secret "samples-operator-tls" not found
Sep 17 17:36:27.831747 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.831722    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fjzlf"]
Sep 17 17:36:27.835851 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.835837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:27.838129 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.838111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:36:27.838433 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.838413    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:36:27.838521 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.838443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ktmh\""
Sep 17 17:36:27.838521 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.838492    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:36:27.838808 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.838794    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:36:27.852184 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.852162    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fjzlf"]
Sep 17 17:36:27.904266 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.904241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-crio-socket\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:27.904359 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.904267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:27.904359 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.904285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:27.904359 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.904348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-data-volume\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:27.904457 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:27.904386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqqp\" (UniqueName: \"kubernetes.io/projected/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-api-access-sdqqp\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.004877 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.004847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-crio-socket\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.004877 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.004874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.005033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.004891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.005033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.004922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-data-volume\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.005033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.004943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqqp\" (UniqueName: \"kubernetes.io/projected/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-api-access-sdqqp\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.005033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.004974    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-crio-socket\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.005033 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.004988    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:28.005226 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.005051    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:28.005226 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.005058    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.505044428 +0000 UTC m=+113.464929679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:28.005226 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.005124    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.505105841 +0000 UTC m=+113.464991103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:28.005327 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.005249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-data-volume\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.017824 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.017800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqqp\" (UniqueName: \"kubernetes.io/projected/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-api-access-sdqqp\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.458373 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.458344    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/1.log"
Sep 17 17:36:28.458756 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.458680    2561 scope.go:117] "RemoveContainer" containerID="697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f"
Sep 17 17:36:28.458851 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.458833    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-mjfcw_openshift-console-operator(f2db503d-1499-4ea7-a841-ab983806a9f6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podUID="f2db503d-1499-4ea7-a841-ab983806a9f6"
Sep 17 17:36:28.508551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.508520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.508551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:28.508553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:28.508796 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.508672    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:28.508796 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.508714    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:29.508702212 +0000 UTC m=+114.468587460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:28.508796 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.508670    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:28.508796 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:28.508796    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:29.508782686 +0000 UTC m=+114.468667933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:29.515922 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:29.515890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:29.515922 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:29.515921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:29.516331 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:29.515996    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:29.516331 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:29.516011    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:29.516331 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:29.516071    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.516054513 +0000 UTC m=+116.475939762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:29.516331 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:29.516087    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.516080559 +0000 UTC m=+116.475965807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:29.799723 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:29.799651    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7qdqz_17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b/dns-node-resolver/0.log"
Sep 17 17:36:30.800044 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:30.800016    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4jfq7_27603f15-420b-4c05-aaa8-fea5ecb33710/node-ca/0.log"
Sep 17 17:36:31.533158 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:31.533122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:31.533158 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:31.533155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:31.533411 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:31.533239    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:31.533411 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:31.533266    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.533411 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:31.533316    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:35.533300607 +0000 UTC m=+120.493185855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:31.533411 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:31.533331    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:35.533325462 +0000 UTC m=+120.493210710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.598659 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:31.598618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bcllq_a94e2c20-8bbd-47d1-b3d1-391b109eccae/serve-healthcheck-canary/0.log"
Sep 17 17:36:31.835246 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:31.835156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:31.835731 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:31.835313    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:36:31.835731 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:31.835393    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls podName:a413e23b-1cf0-43dc-ada7-1f3b06830c5f nodeName:}" failed. No retries permitted until 2026-09-17 17:36:39.835370519 +0000 UTC m=+124.795255769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-49pdh" (UID: "a413e23b-1cf0-43dc-ada7-1f3b06830c5f") : secret "samples-operator-tls" not found
Sep 17 17:36:34.381338 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:34.381296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:34.381748 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:34.381360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:34.381748 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:34.381671    2561 scope.go:117] "RemoveContainer" containerID="697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f"
Sep 17 17:36:34.381829 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:34.381811    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-mjfcw_openshift-console-operator(f2db503d-1499-4ea7-a841-ab983806a9f6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podUID="f2db503d-1499-4ea7-a841-ab983806a9f6"
Sep 17 17:36:35.563939 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:35.563900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:35.563939 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:35.563940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:35.564409 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:35.564020    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:35.564409 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:35.564094    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:43.564073766 +0000 UTC m=+128.523959027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:35.564409 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:35.564123    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:35.564409 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:35.564179    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:43.564167617 +0000 UTC m=+128.524052868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:39.893036 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:39.892989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:39.895269 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:39.895251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a413e23b-1cf0-43dc-ada7-1f3b06830c5f-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-49pdh\" (UID: \"a413e23b-1cf0-43dc-ada7-1f3b06830c5f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:39.977584 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:39.977544    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7cr72\""
Sep 17 17:36:39.984469 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:39.984441    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"
Sep 17 17:36:40.257183 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:40.257152    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh"]
Sep 17 17:36:40.486853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:40.486815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh" event={"ID":"a413e23b-1cf0-43dc-ada7-1f3b06830c5f","Type":"ContainerStarted","Data":"aff354b0d0bf8bb8800e5d08928e31edf8e8445f8ca5b2b7a3367462de057cce"}
Sep 17 17:36:42.495511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:42.495476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh" event={"ID":"a413e23b-1cf0-43dc-ada7-1f3b06830c5f","Type":"ContainerStarted","Data":"34553d78d3f075f64e15e74b3a3ec5babec93620308bc5b65ea02e229000150c"}
Sep 17 17:36:42.495893 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:42.495535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh" event={"ID":"a413e23b-1cf0-43dc-ada7-1f3b06830c5f","Type":"ContainerStarted","Data":"65bef720baab4861fa507a7bddd0b1e402c7e72b015892450605bfe6e6d8f382"}
Sep 17 17:36:42.515581 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:42.515534    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-49pdh" podStartSLOduration=16.87634191 podStartE2EDuration="18.515519112s" totalImagesPullingTime="1.639177202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:42.514724672 +0000 UTC m=+127.474609943" watchObservedRunningTime="2026-09-17 17:36:42.515519112 +0000 UTC m=+127.475404382"
Sep 17 17:36:43.622342 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:43.622303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:43.622342 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:43.622342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:43.622898 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:43.622454    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:43.622898 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:43.622506    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls podName:bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:59.622492857 +0000 UTC m=+144.582378127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fjzlf" (UID: "bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:43.623018 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:43.622979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:46.844853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:46.844803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:36:46.847082 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:46.847060    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/589f88ca-b342-488a-9e88-4c60c80dd6bb-metrics-certs\") pod \"network-metrics-daemon-qdh2w\" (UID: \"589f88ca-b342-488a-9e88-4c60c80dd6bb\") " pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:36:47.137696 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:47.137613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j84n5\""
Sep 17 17:36:47.145305 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:47.145289    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qdh2w"
Sep 17 17:36:47.413247 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:47.413115    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qdh2w"]
Sep 17 17:36:47.415564 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:36:47.415537    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589f88ca_b342_488a_9e88_4c60c80dd6bb.slice/crio-31d2bd221013347502fee0a55a199e50ed12d85da45d09bd0686afe1420bdd66 WatchSource:0}: Error finding container 31d2bd221013347502fee0a55a199e50ed12d85da45d09bd0686afe1420bdd66: Status 404 returned error can't find the container with id 31d2bd221013347502fee0a55a199e50ed12d85da45d09bd0686afe1420bdd66
Sep 17 17:36:47.513056 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:47.513024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qdh2w" event={"ID":"589f88ca-b342-488a-9e88-4c60c80dd6bb","Type":"ContainerStarted","Data":"31d2bd221013347502fee0a55a199e50ed12d85da45d09bd0686afe1420bdd66"}
Sep 17 17:36:48.517759 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:48.517725    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qdh2w" event={"ID":"589f88ca-b342-488a-9e88-4c60c80dd6bb","Type":"ContainerStarted","Data":"6cc8c8fdfd324cc78c121025510404105523ea4be24ce3bad5e942100edd96b9"}
Sep 17 17:36:48.518083 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:48.517776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qdh2w" event={"ID":"589f88ca-b342-488a-9e88-4c60c80dd6bb","Type":"ContainerStarted","Data":"fd437945d45ecd242eca32597ef1c714fdd1c135bc28b0d874c82497e2d3d3b4"}
Sep 17 17:36:48.539375 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:48.539295    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qdh2w" podStartSLOduration=130.67378912 podStartE2EDuration="2m11.539281623s" totalImagesPullingTime="865.492503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:48.537592863 +0000 UTC m=+133.497478136" watchObservedRunningTime="2026-09-17 17:36:48.539281623 +0000 UTC m=+133.499166893"
Sep 17 17:36:49.015145 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.015120    2561 scope.go:117] "RemoveContainer" containerID="697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f"
Sep 17 17:36:49.523089 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.523067    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:36:49.523539 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.523519    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/1.log"
Sep 17 17:36:49.523594 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.523558    2561 generic.go:356] "Generic (PLEG): container finished" podID="f2db503d-1499-4ea7-a841-ab983806a9f6" containerID="79b5e9db14f69f97af2198955f9dd51d49548c6bf586bd42283a8dc1dffa7fc4" exitCode=255
Sep 17 17:36:49.523658 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.523641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" event={"ID":"f2db503d-1499-4ea7-a841-ab983806a9f6","Type":"ContainerDied","Data":"79b5e9db14f69f97af2198955f9dd51d49548c6bf586bd42283a8dc1dffa7fc4"}
Sep 17 17:36:49.523711 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.523694    2561 scope.go:117] "RemoveContainer" containerID="697beebae1f04acf69f66bb74c54394b9b0c29b93b35dd9026d803bb76f8a48f"
Sep 17 17:36:49.524047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:49.524017    2561 scope.go:117] "RemoveContainer" containerID="79b5e9db14f69f97af2198955f9dd51d49548c6bf586bd42283a8dc1dffa7fc4"
Sep 17 17:36:49.524225 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:49.524198    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-mjfcw_openshift-console-operator(f2db503d-1499-4ea7-a841-ab983806a9f6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podUID="f2db503d-1499-4ea7-a841-ab983806a9f6"
Sep 17 17:36:50.527783 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:50.527751    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:36:54.380700 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:54.380669    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:54.381126 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:54.380717    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:36:54.381126 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:54.380986    2561 scope.go:117] "RemoveContainer" containerID="79b5e9db14f69f97af2198955f9dd51d49548c6bf586bd42283a8dc1dffa7fc4"
Sep 17 17:36:54.381221 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:36:54.381123    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-mjfcw_openshift-console-operator(f2db503d-1499-4ea7-a841-ab983806a9f6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podUID="f2db503d-1499-4ea7-a841-ab983806a9f6"
Sep 17 17:36:59.635228 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:59.635184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:59.637552 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:59.637527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fjzlf\" (UID: \"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd\") " pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:59.647803 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:59.647785    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ktmh\""
Sep 17 17:36:59.655353 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:59.655339    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fjzlf"
Sep 17 17:36:59.935065 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:36:59.934989    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fjzlf"]
Sep 17 17:36:59.937859 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:36:59.937829    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1da701_bd7b_4c98_b0d1_f1f7bdf4e4cd.slice/crio-b002200842aa33d58a5b2d5a0048cfe0331b7135b06d3f4134354034f1213e98 WatchSource:0}: Error finding container b002200842aa33d58a5b2d5a0048cfe0331b7135b06d3f4134354034f1213e98: Status 404 returned error can't find the container with id b002200842aa33d58a5b2d5a0048cfe0331b7135b06d3f4134354034f1213e98
Sep 17 17:37:00.561832 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:00.561797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjzlf" event={"ID":"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd","Type":"ContainerStarted","Data":"837c87a5de0b122bb2359a876ef95038b867158c71fd0028d29b8fe2e1877377"}
Sep 17 17:37:00.561999 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:00.561866    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjzlf" event={"ID":"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd","Type":"ContainerStarted","Data":"b002200842aa33d58a5b2d5a0048cfe0331b7135b06d3f4134354034f1213e98"}
Sep 17 17:37:01.566883 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:01.566839    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjzlf" event={"ID":"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd","Type":"ContainerStarted","Data":"5206376025c83f08ff6fb2c14a7eac94cf4bbd455ae9891c2161461493ffad1d"}
Sep 17 17:37:02.048342 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.048296    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"]
Sep 17 17:37:02.053417 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.053393    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"
Sep 17 17:37:02.056459 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.056435    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8s59d\""
Sep 17 17:37:02.069955 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.069932    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"]
Sep 17 17:37:02.089217 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.089185    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"]
Sep 17 17:37:02.092485 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.092464    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.093423 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.093401    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"]
Sep 17 17:37:02.094950 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.094916    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:37:02.095185 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.095165    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:37:02.095300 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.095169    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:37:02.095370 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.095325    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:37:02.096590 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.096571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.099183 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.099137    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-24ww9\""
Sep 17 17:37:02.099291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.099190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:37:02.099291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.099210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:37:02.107297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.107275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"]
Sep 17 17:37:02.109870 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.109849    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"]
Sep 17 17:37:02.157764 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.157729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzljb\" (UniqueName: \"kubernetes.io/projected/30c9e726-45b3-4e6a-b100-ffe935ff0c50-kube-api-access-zzljb\") pod \"network-check-source-7cbc45fddc-7vj4k\" (UID: \"30c9e726-45b3-4e6a-b100-ffe935ff0c50\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"
Sep 17 17:37:02.167111 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.167089    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"]
Sep 17 17:37:02.170419 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.170402    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:02.173103 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.173073    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 17:37:02.173626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.173607    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9dm8p\""
Sep 17 17:37:02.181679 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.181659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"]
Sep 17 17:37:02.258362 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86261ca0-0197-4c31-8a44-462c460d2e01-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-mg9vg\" (UID: \"86261ca0-0197-4c31-8a44-462c460d2e01\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.258362 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bc86174-8e42-4bba-a311-52f5321d6b6e-tmp\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.258570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzljb\" (UniqueName: \"kubernetes.io/projected/30c9e726-45b3-4e6a-b100-ffe935ff0c50-kube-api-access-zzljb\") pod \"network-check-source-7cbc45fddc-7vj4k\" (UID: \"30c9e726-45b3-4e6a-b100-ffe935ff0c50\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"
Sep 17 17:37:02.258570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57dc139b-9e10-4290-869c-4190a55fedb8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2br2j\" (UID: \"57dc139b-9e10-4290-869c-4190a55fedb8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:02.258570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hxn8\" (UniqueName: \"kubernetes.io/projected/0bc86174-8e42-4bba-a311-52f5321d6b6e-kube-api-access-7hxn8\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.258570 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/86261ca0-0197-4c31-8a44-462c460d2e01-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-mg9vg\" (UID: \"86261ca0-0197-4c31-8a44-462c460d2e01\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.258766 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.258624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0bc86174-8e42-4bba-a311-52f5321d6b6e-klusterlet-config\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.269204 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.269183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzljb\" (UniqueName: \"kubernetes.io/projected/30c9e726-45b3-4e6a-b100-ffe935ff0c50-kube-api-access-zzljb\") pod \"network-check-source-7cbc45fddc-7vj4k\" (UID: \"30c9e726-45b3-4e6a-b100-ffe935ff0c50\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"
Sep 17 17:37:02.359924 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.359905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57dc139b-9e10-4290-869c-4190a55fedb8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2br2j\" (UID: \"57dc139b-9e10-4290-869c-4190a55fedb8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:02.360021 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.359948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hxn8\" (UniqueName: \"kubernetes.io/projected/0bc86174-8e42-4bba-a311-52f5321d6b6e-kube-api-access-7hxn8\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.360060 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.360013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/86261ca0-0197-4c31-8a44-462c460d2e01-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-mg9vg\" (UID: \"86261ca0-0197-4c31-8a44-462c460d2e01\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.360091 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.360057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0bc86174-8e42-4bba-a311-52f5321d6b6e-klusterlet-config\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.360091 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.360076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86261ca0-0197-4c31-8a44-462c460d2e01-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-mg9vg\" (UID: \"86261ca0-0197-4c31-8a44-462c460d2e01\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.360170 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.360093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bc86174-8e42-4bba-a311-52f5321d6b6e-tmp\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.360528 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.360508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bc86174-8e42-4bba-a311-52f5321d6b6e-tmp\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.360857 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.360832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86261ca0-0197-4c31-8a44-462c460d2e01-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-mg9vg\" (UID: \"86261ca0-0197-4c31-8a44-462c460d2e01\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.362304 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.362277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0bc86174-8e42-4bba-a311-52f5321d6b6e-klusterlet-config\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.362591 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.362569    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/86261ca0-0197-4c31-8a44-462c460d2e01-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-mg9vg\" (UID: \"86261ca0-0197-4c31-8a44-462c460d2e01\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.362690 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.362586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/57dc139b-9e10-4290-869c-4190a55fedb8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-2br2j\" (UID: \"57dc139b-9e10-4290-869c-4190a55fedb8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:02.363353 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.363340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"
Sep 17 17:37:02.368749 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.368728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hxn8\" (UniqueName: \"kubernetes.io/projected/0bc86174-8e42-4bba-a311-52f5321d6b6e-kube-api-access-7hxn8\") pod \"klusterlet-addon-workmgr-679b99f99d-88rv9\" (UID: \"0bc86174-8e42-4bba-a311-52f5321d6b6e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.405379 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.405348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:02.412027 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.411868    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"
Sep 17 17:37:02.481175 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.480984    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:02.572420 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.572359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fjzlf" event={"ID":"bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd","Type":"ContainerStarted","Data":"2bf58bd025f15f3ba97691a3f629e3e4a5b42c9baabb98bef7830a38c22b0ed5"}
Sep 17 17:37:02.602595 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.602554    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fjzlf" podStartSLOduration=33.358093167 podStartE2EDuration="35.602539758s" totalImagesPullingTime="2.244446591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:36:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:02.601809207 +0000 UTC m=+147.561694479" watchObservedRunningTime="2026-09-17 17:37:02.602539758 +0000 UTC m=+147.562425027"
Sep 17 17:37:02.641422 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.641399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k"]
Sep 17 17:37:02.643459 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:02.643435    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c9e726_45b3_4e6a_b100_ffe935ff0c50.slice/crio-61dae6c518a364cf2441031d05b365e38c83e44ac0d101478f7e523457b31697 WatchSource:0}: Error finding container 61dae6c518a364cf2441031d05b365e38c83e44ac0d101478f7e523457b31697: Status 404 returned error can't find the container with id 61dae6c518a364cf2441031d05b365e38c83e44ac0d101478f7e523457b31697
Sep 17 17:37:02.691228 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.691209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg"]
Sep 17 17:37:02.693752 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:02.693726    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86261ca0_0197_4c31_8a44_462c460d2e01.slice/crio-822095d8ca2e99fb1c204b1974601e78548a3cdc9dec8a53f1efc8aa6f2eb105 WatchSource:0}: Error finding container 822095d8ca2e99fb1c204b1974601e78548a3cdc9dec8a53f1efc8aa6f2eb105: Status 404 returned error can't find the container with id 822095d8ca2e99fb1c204b1974601e78548a3cdc9dec8a53f1efc8aa6f2eb105
Sep 17 17:37:02.696474 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.696451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"]
Sep 17 17:37:02.698848 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:02.698828    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc86174_8e42_4bba_a311_52f5321d6b6e.slice/crio-1a2848a31732e117f190a9be646443b68b1c7494c5693ae8ff9e54ecbd369d58 WatchSource:0}: Error finding container 1a2848a31732e117f190a9be646443b68b1c7494c5693ae8ff9e54ecbd369d58: Status 404 returned error can't find the container with id 1a2848a31732e117f190a9be646443b68b1c7494c5693ae8ff9e54ecbd369d58
Sep 17 17:37:02.753557 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:02.753529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"]
Sep 17 17:37:02.761389 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:02.757453    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dc139b_9e10_4290_869c_4190a55fedb8.slice/crio-6be662f22fdcca2209768fab1e2bf3815e7e8e38041a3e73c89f6d2b56e9b2db WatchSource:0}: Error finding container 6be662f22fdcca2209768fab1e2bf3815e7e8e38041a3e73c89f6d2b56e9b2db: Status 404 returned error can't find the container with id 6be662f22fdcca2209768fab1e2bf3815e7e8e38041a3e73c89f6d2b56e9b2db
Sep 17 17:37:03.577747 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:03.577703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j" event={"ID":"57dc139b-9e10-4290-869c-4190a55fedb8","Type":"ContainerStarted","Data":"6be662f22fdcca2209768fab1e2bf3815e7e8e38041a3e73c89f6d2b56e9b2db"}
Sep 17 17:37:03.579340 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:03.579262    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg" event={"ID":"86261ca0-0197-4c31-8a44-462c460d2e01","Type":"ContainerStarted","Data":"822095d8ca2e99fb1c204b1974601e78548a3cdc9dec8a53f1efc8aa6f2eb105"}
Sep 17 17:37:03.580827 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:03.580802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9" event={"ID":"0bc86174-8e42-4bba-a311-52f5321d6b6e","Type":"ContainerStarted","Data":"1a2848a31732e117f190a9be646443b68b1c7494c5693ae8ff9e54ecbd369d58"}
Sep 17 17:37:03.583138 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:03.583060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k" event={"ID":"30c9e726-45b3-4e6a-b100-ffe935ff0c50","Type":"ContainerStarted","Data":"29ec673cb47e1c74a31f56616f31a83d3759c099db6c80183b106ca3ddab6ca5"}
Sep 17 17:37:03.583138 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:03.583113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k" event={"ID":"30c9e726-45b3-4e6a-b100-ffe935ff0c50","Type":"ContainerStarted","Data":"61dae6c518a364cf2441031d05b365e38c83e44ac0d101478f7e523457b31697"}
Sep 17 17:37:03.612795 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:03.612737    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7vj4k" podStartSLOduration=1.612718678 podStartE2EDuration="1.612718678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:03.611705669 +0000 UTC m=+148.571590938" watchObservedRunningTime="2026-09-17 17:37:03.612718678 +0000 UTC m=+148.572603959"
Sep 17 17:37:04.589985 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.589947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j" event={"ID":"57dc139b-9e10-4290-869c-4190a55fedb8","Type":"ContainerStarted","Data":"891089cc95556e2f874c8da3c023a0d625e4323ead316445f35de0b780d06736"}
Sep 17 17:37:04.590457 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.590417    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:04.593185 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.593156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg" event={"ID":"86261ca0-0197-4c31-8a44-462c460d2e01","Type":"ContainerStarted","Data":"59d9bb7af726427eb8074425c9d22b7894972afff62b20d2d71123db1ea4bda0"}
Sep 17 17:37:04.596592 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.596568    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j"
Sep 17 17:37:04.613900 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.613855    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-2br2j" podStartSLOduration=1.1810170150000001 podStartE2EDuration="2.613840944s" totalImagesPullingTime="1.432823929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:04.611948081 +0000 UTC m=+149.571833351" watchObservedRunningTime="2026-09-17 17:37:04.613840944 +0000 UTC m=+149.573726213"
Sep 17 17:37:04.669520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.669467    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-mg9vg" podStartSLOduration=1.172294382 podStartE2EDuration="2.669444651s" totalImagesPullingTime="1.497150269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:04.668378853 +0000 UTC m=+149.628264127" watchObservedRunningTime="2026-09-17 17:37:04.669444651 +0000 UTC m=+149.629329925"
Sep 17 17:37:04.904871 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.904787    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"]
Sep 17 17:37:04.908531 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.908498    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:04.918064 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.918038    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 17:37:04.918247 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.918045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 17:37:04.919102 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.918844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:37:04.919102 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.918964    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:37:04.919102 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.918844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:37:04.919328 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.919233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8q5dz\""
Sep 17 17:37:04.930828 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:04.930548    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"]
Sep 17 17:37:05.082530 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.082493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.082709 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.082648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.082781 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.082711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.082781 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.082739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7625\" (UniqueName: \"kubernetes.io/projected/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-kube-api-access-s7625\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.184091 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.184018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.184254 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.184182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.189465 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.184644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.189465 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.184704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7625\" (UniqueName: \"kubernetes.io/projected/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-kube-api-access-s7625\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.189465 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:05.185154    2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 17:37:05.189465 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:05.185222    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-tls podName:dd6b2f1d-6863-45b9-9f6d-29d791ad745a nodeName:}" failed. No retries permitted until 2026-09-17 17:37:05.685202887 +0000 UTC m=+150.645088156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-bwl9k" (UID: "dd6b2f1d-6863-45b9-9f6d-29d791ad745a") : secret "prometheus-operator-tls" not found
Sep 17 17:37:05.189465 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.185539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.190368 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.190340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.194580 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.194552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7625\" (UniqueName: \"kubernetes.io/projected/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-kube-api-access-s7625\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.688423 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.688354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.691038 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.691013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd6b2f1d-6863-45b9-9f6d-29d791ad745a-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-bwl9k\" (UID: \"dd6b2f1d-6863-45b9-9f6d-29d791ad745a\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:05.820310 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:05.820274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"
Sep 17 17:37:07.108920 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:07.108853    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k"]
Sep 17 17:37:07.112710 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:07.112683    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6b2f1d_6863_45b9_9f6d_29d791ad745a.slice/crio-fe125bcd42006d6e76184663540a6a680f4f21cdf4b6ed56ce5613bc02ee9b5e WatchSource:0}: Error finding container fe125bcd42006d6e76184663540a6a680f4f21cdf4b6ed56ce5613bc02ee9b5e: Status 404 returned error can't find the container with id fe125bcd42006d6e76184663540a6a680f4f21cdf4b6ed56ce5613bc02ee9b5e
Sep 17 17:37:07.607389 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:07.607353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9" event={"ID":"0bc86174-8e42-4bba-a311-52f5321d6b6e","Type":"ContainerStarted","Data":"ad7d66058424e7971c8b5603e954a2061a2653b40ecfb0ce68758dfd9eacf0d1"}
Sep 17 17:37:07.607575 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:07.607541    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:07.608713 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:07.608687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k" event={"ID":"dd6b2f1d-6863-45b9-9f6d-29d791ad745a","Type":"ContainerStarted","Data":"fe125bcd42006d6e76184663540a6a680f4f21cdf4b6ed56ce5613bc02ee9b5e"}
Sep 17 17:37:07.609447 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:07.609429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9"
Sep 17 17:37:07.627200 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:07.627155    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679b99f99d-88rv9" podStartSLOduration=1.414996187 podStartE2EDuration="5.627141098s" totalImagesPullingTime="4.212144911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:07.62643457 +0000 UTC m=+152.586319839" watchObservedRunningTime="2026-09-17 17:37:07.627141098 +0000 UTC m=+152.587026368"
Sep 17 17:37:08.614358 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:08.614322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k" event={"ID":"dd6b2f1d-6863-45b9-9f6d-29d791ad745a","Type":"ContainerStarted","Data":"df25df3ad1bc0eed1e2e322e408f41005fefcd0864bef170b8d0b5ccce9e1338"}
Sep 17 17:37:08.614756 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:08.614408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k" event={"ID":"dd6b2f1d-6863-45b9-9f6d-29d791ad745a","Type":"ContainerStarted","Data":"973414d25437bc5a35e9398a4d2b3f1c14aa092536477792ef18a20f31b439e9"}
Sep 17 17:37:08.636541 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:08.636447    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-bwl9k" podStartSLOduration=3.384594184 podStartE2EDuration="4.636431019s" totalImagesPullingTime="1.251836835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:08.635459772 +0000 UTC m=+153.595345044" watchObservedRunningTime="2026-09-17 17:37:08.636431019 +0000 UTC m=+153.596316288"
Sep 17 17:37:09.018557 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:09.018529    2561 scope.go:117] "RemoveContainer" containerID="79b5e9db14f69f97af2198955f9dd51d49548c6bf586bd42283a8dc1dffa7fc4"
Sep 17 17:37:09.018747 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:09.018705    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-mjfcw_openshift-console-operator(f2db503d-1499-4ea7-a841-ab983806a9f6)\"" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podUID="f2db503d-1499-4ea7-a841-ab983806a9f6"
Sep 17 17:37:10.496010 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.495981    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jv69z"]
Sep 17 17:37:10.499297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.499281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.502354 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.502335    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:37:10.502687 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.502657    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:37:10.502774 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.502691    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:37:10.502774 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.502694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kjs7l\""
Sep 17 17:37:10.627413 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-sys\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-wtmp\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-textfile\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28070dd8-00c3-4f8e-b471-17e7ee28253d-metrics-client-ca\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627837 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-tls\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627837 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clp2\" (UniqueName: \"kubernetes.io/projected/28070dd8-00c3-4f8e-b471-17e7ee28253d-kube-api-access-6clp2\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627837 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-root\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.627837 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.627650    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728577 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-sys\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728600    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-wtmp\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-textfile\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28070dd8-00c3-4f8e-b471-17e7ee28253d-metrics-client-ca\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-sys\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.728765 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-tls\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clp2\" (UniqueName: \"kubernetes.io/projected/28070dd8-00c3-4f8e-b471-17e7ee28253d-kube-api-access-6clp2\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-root\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728848    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-wtmp\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/28070dd8-00c3-4f8e-b471-17e7ee28253d-root\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.728914    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729045 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.729031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-textfile\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729379 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.729357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28070dd8-00c3-4f8e-b471-17e7ee28253d-metrics-client-ca\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.729443 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.729412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-accelerators-collector-config\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.731016 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.730995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.731119 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.731095    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/28070dd8-00c3-4f8e-b471-17e7ee28253d-node-exporter-tls\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.738107 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.738088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clp2\" (UniqueName: \"kubernetes.io/projected/28070dd8-00c3-4f8e-b471-17e7ee28253d-kube-api-access-6clp2\") pod \"node-exporter-jv69z\" (UID: \"28070dd8-00c3-4f8e-b471-17e7ee28253d\") " pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.807927 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:10.807864    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jv69z"
Sep 17 17:37:10.818784 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:10.818760    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28070dd8_00c3_4f8e_b471_17e7ee28253d.slice/crio-cb98a23c30b8feb5cff8fdb5663af83d116680b8700a5bd203a9b4e21f97122b WatchSource:0}: Error finding container cb98a23c30b8feb5cff8fdb5663af83d116680b8700a5bd203a9b4e21f97122b: Status 404 returned error can't find the container with id cb98a23c30b8feb5cff8fdb5663af83d116680b8700a5bd203a9b4e21f97122b
Sep 17 17:37:11.625813 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:11.625780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jv69z" event={"ID":"28070dd8-00c3-4f8e-b471-17e7ee28253d","Type":"ContainerStarted","Data":"cb98a23c30b8feb5cff8fdb5663af83d116680b8700a5bd203a9b4e21f97122b"}
Sep 17 17:37:12.630667 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:12.630611    2561 generic.go:356] "Generic (PLEG): container finished" podID="28070dd8-00c3-4f8e-b471-17e7ee28253d" containerID="6d6b845e23016bf72ad7217b9610321874795d1c50bccf062d4fc9e21a35dae8" exitCode=0
Sep 17 17:37:12.631109 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:12.630701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jv69z" event={"ID":"28070dd8-00c3-4f8e-b471-17e7ee28253d","Type":"ContainerDied","Data":"6d6b845e23016bf72ad7217b9610321874795d1c50bccf062d4fc9e21a35dae8"}
Sep 17 17:37:13.139552 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:13.139512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02"
Sep 17 17:37:13.182432 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:13.182405    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vssft" podUID="cc9faa45-7dfc-4569-9c1f-30a623d562e7"
Sep 17 17:37:13.636033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:13.636005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:37:13.636033 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:13.635998    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jv69z" event={"ID":"28070dd8-00c3-4f8e-b471-17e7ee28253d","Type":"ContainerStarted","Data":"0304474a94f17555adbe844ec46fcd697493ac08fb80bfd691d92fead4ae0d9c"}
Sep 17 17:37:13.636496 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:13.636082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jv69z" event={"ID":"28070dd8-00c3-4f8e-b471-17e7ee28253d","Type":"ContainerStarted","Data":"29252bcbd5d3ab98aa7add61f247aa696a43b13229ff2bc0b3d849b3d01a393a"}
Sep 17 17:37:13.659015 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:13.658967    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jv69z" podStartSLOduration=2.48353523 podStartE2EDuration="3.658953676s" totalImagesPullingTime="1.116108089s" totalInitContainerRuntime="59.310357ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:13.658916006 +0000 UTC m=+158.618801286" watchObservedRunningTime="2026-09-17 17:37:13.658953676 +0000 UTC m=+158.618838925"
Sep 17 17:37:14.751355 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.751325    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"]
Sep 17 17:37:14.754418 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.754401    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.763855 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.763835    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:37:14.763949 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.763873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4sfa3rvqj3qqt\""
Sep 17 17:37:14.764138 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.764124    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:37:14.765207 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.765191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-w797g\""
Sep 17 17:37:14.765305 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.765212    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:37:14.765305 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.765279    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:37:14.770691 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.770671    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"]
Sep 17 17:37:14.863972 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.863938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-secret-metrics-server-client-certs\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.863972 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.863972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca590260-399a-4d1e-a0bf-52744c163003-metrics-server-audit-profiles\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.864157 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.863992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-secret-metrics-server-tls\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.864157 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.864041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca590260-399a-4d1e-a0bf-52744c163003-audit-log\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.864157 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.864085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca590260-399a-4d1e-a0bf-52744c163003-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.864157 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.864104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-client-ca-bundle\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.864157 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.864133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzr4\" (UniqueName: \"kubernetes.io/projected/ca590260-399a-4d1e-a0bf-52744c163003-kube-api-access-nzzr4\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964446 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-secret-metrics-server-client-certs\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca590260-399a-4d1e-a0bf-52744c163003-metrics-server-audit-profiles\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-secret-metrics-server-tls\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964488    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca590260-399a-4d1e-a0bf-52744c163003-audit-log\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca590260-399a-4d1e-a0bf-52744c163003-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-client-ca-bundle\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzr4\" (UniqueName: \"kubernetes.io/projected/ca590260-399a-4d1e-a0bf-52744c163003-kube-api-access-nzzr4\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.964972 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.964954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca590260-399a-4d1e-a0bf-52744c163003-audit-log\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.965381 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.965355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca590260-399a-4d1e-a0bf-52744c163003-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.965476 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.965438    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca590260-399a-4d1e-a0bf-52744c163003-metrics-server-audit-profiles\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.967009 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.966983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-secret-metrics-server-client-certs\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.967119 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.967080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-secret-metrics-server-tls\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.967119 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.967099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca590260-399a-4d1e-a0bf-52744c163003-client-ca-bundle\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:14.974508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:14.974489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzr4\" (UniqueName: \"kubernetes.io/projected/ca590260-399a-4d1e-a0bf-52744c163003-kube-api-access-nzzr4\") pod \"metrics-server-5c84f6d5c6-x2sw2\" (UID: \"ca590260-399a-4d1e-a0bf-52744c163003\") " pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:15.050816 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.050740    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"]
Sep 17 17:37:15.054910 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.054896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.057898 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.057875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 17:37:15.058243 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.058223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5vpvp\""
Sep 17 17:37:15.063544 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.063524    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:15.065660 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.065641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"]
Sep 17 17:37:15.166293 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.166261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nj29\" (UniqueName: \"kubernetes.io/projected/9a015893-311e-427b-b258-2fd5d85e6a25-kube-api-access-9nj29\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.166421 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.166398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a015893-311e-427b-b258-2fd5d85e6a25-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.267296 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.267265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a015893-311e-427b-b258-2fd5d85e6a25-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.267416 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.267307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nj29\" (UniqueName: \"kubernetes.io/projected/9a015893-311e-427b-b258-2fd5d85e6a25-kube-api-access-9nj29\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.267416 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:15.267393    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 17:37:15.267502 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:37:15.267454    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a015893-311e-427b-b258-2fd5d85e6a25-monitoring-plugin-cert podName:9a015893-311e-427b-b258-2fd5d85e6a25 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:15.767437106 +0000 UTC m=+160.727322354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9a015893-311e-427b-b258-2fd5d85e6a25-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-q8z58" (UID: "9a015893-311e-427b-b258-2fd5d85e6a25") : secret "monitoring-plugin-cert" not found
Sep 17 17:37:15.277921 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.277897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nj29\" (UniqueName: \"kubernetes.io/projected/9a015893-311e-427b-b258-2fd5d85e6a25-kube-api-access-9nj29\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.339221 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.339153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"]
Sep 17 17:37:15.342800 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:15.342778    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca590260_399a_4d1e_a0bf_52744c163003.slice/crio-f0bd73bba4337006ae91a9ec61ec4c7fcbcfdf721a1baa777b84a4a29ebb61c9 WatchSource:0}: Error finding container f0bd73bba4337006ae91a9ec61ec4c7fcbcfdf721a1baa777b84a4a29ebb61c9: Status 404 returned error can't find the container with id f0bd73bba4337006ae91a9ec61ec4c7fcbcfdf721a1baa777b84a4a29ebb61c9
Sep 17 17:37:15.634890 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.634810    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d689769bd-zw5n4"]
Sep 17 17:37:15.639387 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.639369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.642294 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.642273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fcb59\""
Sep 17 17:37:15.642399 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.642317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 17:37:15.642399 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.642334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 17:37:15.642659 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.642618    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 17:37:15.642789 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.642773    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 17:37:15.642920 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.642901    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 17:37:15.644197 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.644166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2" event={"ID":"ca590260-399a-4d1e-a0bf-52744c163003","Type":"ContainerStarted","Data":"f0bd73bba4337006ae91a9ec61ec4c7fcbcfdf721a1baa777b84a4a29ebb61c9"}
Sep 17 17:37:15.651605 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.651562    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 17:37:15.654186 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.654164    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d689769bd-zw5n4"]
Sep 17 17:37:15.669455 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669565 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669565 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-secret-telemeter-client\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-telemeter-client-tls\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669610    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-serving-certs-ca-bundle\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669661    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-federate-client-tls\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669774 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jdx\" (UniqueName: \"kubernetes.io/projected/c101df20-1d1c-447d-8d81-dea041093e94-kube-api-access-47jdx\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.669774 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.669699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-metrics-client-ca\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.770910 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.770883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-secret-telemeter-client\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.770918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-telemeter-client-tls\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.770941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-serving-certs-ca-bundle\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.770963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-federate-client-tls\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.770981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jdx\" (UniqueName: \"kubernetes.io/projected/c101df20-1d1c-447d-8d81-dea041093e94-kube-api-access-47jdx\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.771094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-metrics-client-ca\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.771225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.771275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a015893-311e-427b-b258-2fd5d85e6a25-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.771361 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.771302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.771836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.771813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-serving-certs-ca-bundle\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.772051 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.772005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-metrics-client-ca\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.772291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.772270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c101df20-1d1c-447d-8d81-dea041093e94-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.773733 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.773701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-secret-telemeter-client\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.774162 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.774142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-federate-client-tls\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.774239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.774195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a015893-311e-427b-b258-2fd5d85e6a25-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-q8z58\" (UID: \"9a015893-311e-427b-b258-2fd5d85e6a25\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:15.774301 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.774250    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.774301 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.774281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c101df20-1d1c-447d-8d81-dea041093e94-telemeter-client-tls\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.780101 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.780081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jdx\" (UniqueName: \"kubernetes.io/projected/c101df20-1d1c-447d-8d81-dea041093e94-kube-api-access-47jdx\") pod \"telemeter-client-d689769bd-zw5n4\" (UID: \"c101df20-1d1c-447d-8d81-dea041093e94\") " pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.954061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.953988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4"
Sep 17 17:37:15.964908 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:15.964892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:16.237229 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.236626    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d689769bd-zw5n4"]
Sep 17 17:37:16.239614 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:16.239577    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc101df20_1d1c_447d_8d81_dea041093e94.slice/crio-73c5ea367478e743f5e07139bf3991e5dfea71fafaf00fc6d8c8912bbc27d367 WatchSource:0}: Error finding container 73c5ea367478e743f5e07139bf3991e5dfea71fafaf00fc6d8c8912bbc27d367: Status 404 returned error can't find the container with id 73c5ea367478e743f5e07139bf3991e5dfea71fafaf00fc6d8c8912bbc27d367
Sep 17 17:37:16.250896 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.250873    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"]
Sep 17 17:37:16.253623 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:16.253600    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a015893_311e_427b_b258_2fd5d85e6a25.slice/crio-a308de10426d1638dc89bc28e3ea3bb8ad9be27efa7c417093fc7b907b63444e WatchSource:0}: Error finding container a308de10426d1638dc89bc28e3ea3bb8ad9be27efa7c417093fc7b907b63444e: Status 404 returned error can't find the container with id a308de10426d1638dc89bc28e3ea3bb8ad9be27efa7c417093fc7b907b63444e
Sep 17 17:37:16.649007 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.648916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58" event={"ID":"9a015893-311e-427b-b258-2fd5d85e6a25","Type":"ContainerStarted","Data":"a308de10426d1638dc89bc28e3ea3bb8ad9be27efa7c417093fc7b907b63444e"}
Sep 17 17:37:16.650329 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.650302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4" event={"ID":"c101df20-1d1c-447d-8d81-dea041093e94","Type":"ContainerStarted","Data":"73c5ea367478e743f5e07139bf3991e5dfea71fafaf00fc6d8c8912bbc27d367"}
Sep 17 17:37:16.775269 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.775232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:16.780927 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.780899    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.784031 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.784005    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:37:16.784175 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.784117    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1cirbv1237r1j\""
Sep 17 17:37:16.786714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.785216    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:37:16.786714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.785526    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:37:16.786714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.785729    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:37:16.786714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.786049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:37:16.786714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.786362    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:37:16.786714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.786551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hlvrj\""
Sep 17 17:37:16.787103 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.786759    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:37:16.787103 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.787014    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:37:16.787231 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.787211    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:37:16.787343 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.787276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:37:16.787428 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.787419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:37:16.787621 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.787604    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:37:16.789347 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.789329    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:37:16.791652 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.791491    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:37:16.799757 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.799735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:16.882215 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882215 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qh9\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-kube-api-access-j6qh9\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882689 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882689 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882689 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882689 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882689 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-config\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.882853 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.883171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.882860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984154 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984154 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qh9\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-kube-api-access-j6qh9\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.984666 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-config\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.985254 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.985254 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.985254 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.984729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.985446 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.985422    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.986754 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.985983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.986754 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.986100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.986754 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.986207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.986754 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.986611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.987836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.987599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.989432 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.988300    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.989432 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.989391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.991949 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.991922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.997397 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.997351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.998087 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.997747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.998087 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.997994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.998087 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.998050    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.998297 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.998158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.998599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.998558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qh9\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-kube-api-access-j6qh9\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.999209 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.999173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.999293 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.999262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-config\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:16.999925 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.999901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:17.000124 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:17.000078    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:17.001016 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:16.999732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:17.095345 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:17.095309    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:17.411617 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:17.411415    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:37:17.414554 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:17.414523    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95c072e_b110_42ad_9977_e1b92e39f9d6.slice/crio-b9cf256c14aa5d8e39c917634c9df00d6e931b381b6be3b0c7501c03aca3171f WatchSource:0}: Error finding container b9cf256c14aa5d8e39c917634c9df00d6e931b381b6be3b0c7501c03aca3171f: Status 404 returned error can't find the container with id b9cf256c14aa5d8e39c917634c9df00d6e931b381b6be3b0c7501c03aca3171f
Sep 17 17:37:17.656344 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:17.656308    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2" event={"ID":"ca590260-399a-4d1e-a0bf-52744c163003","Type":"ContainerStarted","Data":"cefdf902ecb7a766fc4a7c78d9fa2d9bb4d01b183f0e8833a5126f9642a8d1e9"}
Sep 17 17:37:17.657696 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:17.657660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"b9cf256c14aa5d8e39c917634c9df00d6e931b381b6be3b0c7501c03aca3171f"}
Sep 17 17:37:17.677279 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:17.676923    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2" podStartSLOduration=2.117745618 podStartE2EDuration="3.676892776s" totalImagesPullingTime="1.559147158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:17.676574543 +0000 UTC m=+162.636459829" watchObservedRunningTime="2026-09-17 17:37:17.676892776 +0000 UTC m=+162.636778048"
Sep 17 17:37:18.095470 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.095426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:37:18.095919 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.095494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:37:18.098719 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.098660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cc9faa45-7dfc-4569-9c1f-30a623d562e7-metrics-tls\") pod \"dns-default-vssft\" (UID: \"cc9faa45-7dfc-4569-9c1f-30a623d562e7\") " pod="openshift-dns/dns-default-vssft"
Sep 17 17:37:18.098719 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.098673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"image-registry-cbfd45bdf-c6frm\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") " pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:37:18.139721 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.139689    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-55z8g\""
Sep 17 17:37:18.147215 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.147196    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:37:18.933388 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:18.933296    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbfd45bdf-c6frm"]
Sep 17 17:37:18.936529 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:18.936502    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19b9e24_40a5_4a33_8f8e_24a299fb0f02.slice/crio-097484b8fb4b577add9d65d544833879bfb0dfe3cf206d34540c056a1e306c15 WatchSource:0}: Error finding container 097484b8fb4b577add9d65d544833879bfb0dfe3cf206d34540c056a1e306c15: Status 404 returned error can't find the container with id 097484b8fb4b577add9d65d544833879bfb0dfe3cf206d34540c056a1e306c15
Sep 17 17:37:19.670551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.670510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" event={"ID":"f19b9e24-40a5-4a33-8f8e-24a299fb0f02","Type":"ContainerStarted","Data":"a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3"}
Sep 17 17:37:19.670954 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.670587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" event={"ID":"f19b9e24-40a5-4a33-8f8e-24a299fb0f02","Type":"ContainerStarted","Data":"097484b8fb4b577add9d65d544833879bfb0dfe3cf206d34540c056a1e306c15"}
Sep 17 17:37:19.670954 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.670615    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:37:19.672159 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.672138    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad" exitCode=0
Sep 17 17:37:19.672267 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.672220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad"}
Sep 17 17:37:19.673680 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.673655    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58" event={"ID":"9a015893-311e-427b-b258-2fd5d85e6a25","Type":"ContainerStarted","Data":"1648e20978d2b35e6fd35ac91846d33f80f3e2584bd55981a4f53149a3a0f39f"}
Sep 17 17:37:19.673875 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.673803    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:19.675919 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.675897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4" event={"ID":"c101df20-1d1c-447d-8d81-dea041093e94","Type":"ContainerStarted","Data":"19d1c2eb2ed334f2aa54a8265cd3d6bda5e40c0c8bbf44c7ad69162f10ad9bd4"}
Sep 17 17:37:19.675971 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.675948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4" event={"ID":"c101df20-1d1c-447d-8d81-dea041093e94","Type":"ContainerStarted","Data":"3bd942578295ba9a59aad4e2bae630196e15a2fd5e77c992ffecdbfb686f609b"}
Sep 17 17:37:19.676019 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.675974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4" event={"ID":"c101df20-1d1c-447d-8d81-dea041093e94","Type":"ContainerStarted","Data":"406dfcd95ffb99a4005e149ab0172f622496595109ee925a61972be908321c1b"}
Sep 17 17:37:19.678892 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.678871    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58"
Sep 17 17:37:19.719274 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.719240    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" podStartSLOduration=162.719228504 podStartE2EDuration="2m42.719228504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:19.712186931 +0000 UTC m=+164.672072200" watchObservedRunningTime="2026-09-17 17:37:19.719228504 +0000 UTC m=+164.679113768"
Sep 17 17:37:19.775993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.775953    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-q8z58" podStartSLOduration=2.390881291 podStartE2EDuration="4.775938904s" totalImagesPullingTime="2.385057613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:19.77539537 +0000 UTC m=+164.735280641" watchObservedRunningTime="2026-09-17 17:37:19.775938904 +0000 UTC m=+164.735824234"
Sep 17 17:37:19.892350 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:19.892297    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d689769bd-zw5n4" podStartSLOduration=2.481876889 podStartE2EDuration="4.892282306s" totalImagesPullingTime="2.410405417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:19.892197038 +0000 UTC m=+164.852082308" watchObservedRunningTime="2026-09-17 17:37:19.892282306 +0000 UTC m=+164.852167572"
Sep 17 17:37:20.014269 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:20.014238    2561 scope.go:117] "RemoveContainer" containerID="79b5e9db14f69f97af2198955f9dd51d49548c6bf586bd42283a8dc1dffa7fc4"
Sep 17 17:37:20.684381 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:20.684352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:37:20.684830 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:20.684552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" event={"ID":"f2db503d-1499-4ea7-a841-ab983806a9f6","Type":"ContainerStarted","Data":"8d1a617f58477016d363a74cc8a6cd1be5342a17214287c6da8c05c3ad0f92c8"}
Sep 17 17:37:20.685726 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:20.685292    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:37:20.711192 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:20.711142    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw" podStartSLOduration=55.132806033 podStartE2EDuration="56.711124862s" totalImagesPullingTime="1.578318829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:20.70956442 +0000 UTC m=+165.669449693" watchObservedRunningTime="2026-09-17 17:37:20.711124862 +0000 UTC m=+165.671010135"
Sep 17 17:37:21.047238 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:21.047213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-mjfcw"
Sep 17 17:37:23.702777 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:23.702739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e"}
Sep 17 17:37:23.703145 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:23.702827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697"}
Sep 17 17:37:24.710597 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:24.710568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241"}
Sep 17 17:37:25.717788 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:25.717749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d"}
Sep 17 17:37:25.718169 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:25.717815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83"}
Sep 17 17:37:25.718169 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:25.717840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerStarted","Data":"771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07"}
Sep 17 17:37:25.754575 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:25.754524    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.6618170340000002 podStartE2EDuration="9.754506974s" totalImagesPullingTime="6.070119117s" totalInitContainerRuntime="22.570823ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:25.751733029 +0000 UTC m=+170.711618298" watchObservedRunningTime="2026-09-17 17:37:25.754506974 +0000 UTC m=+170.714392243"
Sep 17 17:37:27.016391 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:27.016360    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vssft"
Sep 17 17:37:27.019351 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:27.019330    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dng7x\""
Sep 17 17:37:27.026773 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:27.026757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vssft"
Sep 17 17:37:27.096834 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:27.096796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:37:27.300439 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:27.300413    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vssft"]
Sep 17 17:37:27.303025 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:37:27.302997    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9faa45_7dfc_4569_9c1f_30a623d562e7.slice/crio-06f2f5f278b3097f1c6cd165dbbffaa964607fe9b6af374bb997e349b73fbcf4 WatchSource:0}: Error finding container 06f2f5f278b3097f1c6cd165dbbffaa964607fe9b6af374bb997e349b73fbcf4: Status 404 returned error can't find the container with id 06f2f5f278b3097f1c6cd165dbbffaa964607fe9b6af374bb997e349b73fbcf4
Sep 17 17:37:27.727301 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:27.727179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vssft" event={"ID":"cc9faa45-7dfc-4569-9c1f-30a623d562e7","Type":"ContainerStarted","Data":"06f2f5f278b3097f1c6cd165dbbffaa964607fe9b6af374bb997e349b73fbcf4"}
Sep 17 17:37:29.737848 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:29.737811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vssft" event={"ID":"cc9faa45-7dfc-4569-9c1f-30a623d562e7","Type":"ContainerStarted","Data":"23139e1feb54e6ccab5b14bf65f3e68d646b49486ad97a653ee941c7af653dae"}
Sep 17 17:37:29.738272 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:29.737872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vssft" event={"ID":"cc9faa45-7dfc-4569-9c1f-30a623d562e7","Type":"ContainerStarted","Data":"2c28c092f911c7e9943222f7512e2716fbeadc6a5cb39a72c91e3731dc14aa12"}
Sep 17 17:37:29.738272 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:29.737933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vssft"
Sep 17 17:37:29.760581 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:29.760536    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vssft" podStartSLOduration=138.30119924 podStartE2EDuration="2m19.760522236s" totalImagesPullingTime="1.459322996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:29.759570709 +0000 UTC m=+174.719455982" watchObservedRunningTime="2026-09-17 17:37:29.760522236 +0000 UTC m=+174.720407507"
Sep 17 17:37:35.063755 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:35.063719    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:35.064331 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:35.063781    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:38.151257 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:38.151223    2561 patch_prober.go:28] interesting pod/image-registry-cbfd45bdf-c6frm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:38.151606 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:38.151275    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:39.743697 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:39.743670    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vssft"
Sep 17 17:37:40.689717 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:40.689689    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:37:52.572121 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:52.572082    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbfd45bdf-c6frm"]
Sep 17 17:37:55.069446 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:55.069416    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:37:55.074684 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:37:55.074661    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c84f6d5c6-x2sw2"
Sep 17 17:38:17.096401 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:17.096359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:17.120730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:17.120702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:17.592721 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:17.592679    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" containerName="registry" containerID="cri-o://a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3" gracePeriod=30
Sep 17 17:38:17.952967 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:17.952892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:18.832920 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.832895    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:38:18.905137 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-trusted-ca\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-ca-trust-extracted\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-bound-sa-token\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905217    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4n6s\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-kube-api-access-r4n6s\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-installation-pull-secrets\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905291 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-image-registry-private-configuration\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905552 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905485    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-certificates\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905552 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905544    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") pod \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\" (UID: \"f19b9e24-40a5-4a33-8f8e-24a299fb0f02\") "
Sep 17 17:38:18.905664 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905577    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:18.906104 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905821    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-trusted-ca\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:18.906104 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.905920    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:18.908127 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.908077    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:18.908242 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.908139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:18.908242 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.908164    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:18.908354 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.908262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-kube-api-access-r4n6s" (OuterVolumeSpecName: "kube-api-access-r4n6s") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "kube-api-access-r4n6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:18.908599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.908579    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:18.913779 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.913757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f19b9e24-40a5-4a33-8f8e-24a299fb0f02" (UID: "f19b9e24-40a5-4a33-8f8e-24a299fb0f02"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:18.941980 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.941926    2561 generic.go:356] "Generic (PLEG): container finished" podID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" containerID="a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3" exitCode=0
Sep 17 17:38:18.942056 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.941977    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm"
Sep 17 17:38:18.942056 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.942011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" event={"ID":"f19b9e24-40a5-4a33-8f8e-24a299fb0f02","Type":"ContainerDied","Data":"a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3"}
Sep 17 17:38:18.942123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.942084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbfd45bdf-c6frm" event={"ID":"f19b9e24-40a5-4a33-8f8e-24a299fb0f02","Type":"ContainerDied","Data":"097484b8fb4b577add9d65d544833879bfb0dfe3cf206d34540c056a1e306c15"}
Sep 17 17:38:18.942123 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.942106    2561 scope.go:117] "RemoveContainer" containerID="a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3"
Sep 17 17:38:18.952822 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.952802    2561 scope.go:117] "RemoveContainer" containerID="a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3"
Sep 17 17:38:18.953119 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:18.953099    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3\": container with ID starting with a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3 not found: ID does not exist" containerID="a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3"
Sep 17 17:38:18.953194 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.953127    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3"} err="failed to get container status \"a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3\": rpc error: code = NotFound desc = could not find container \"a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3\": container with ID starting with a299324357e3b123341f7fe6b533bbad7d25887f03061f80155860e7bf8bf4c3 not found: ID does not exist"
Sep 17 17:38:18.968105 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.968083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbfd45bdf-c6frm"]
Sep 17 17:38:18.972955 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:18.972937    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cbfd45bdf-c6frm"]
Sep 17 17:38:19.007212 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007188    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-bound-sa-token\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.007306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007293    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4n6s\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-kube-api-access-r4n6s\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.007306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007305    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-installation-pull-secrets\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.007439 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007315    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-image-registry-private-configuration\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.007439 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007323    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-certificates\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.007439 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007333    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-registry-tls\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.007439 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.007341    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f19b9e24-40a5-4a33-8f8e-24a299fb0f02-ca-trust-extracted\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:19.019039 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:19.019015    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" path="/var/lib/kubelet/pods/f19b9e24-40a5-4a33-8f8e-24a299fb0f02/volumes"
Sep 17 17:38:35.184505 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.184468    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:38:35.185010 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.184828    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d" gracePeriod=600
Sep 17 17:38:35.185010 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.184876    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy" containerID="cri-o://344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83" gracePeriod=600
Sep 17 17:38:35.185010 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.184866    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="thanos-sidecar" containerID="cri-o://ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241" gracePeriod=600
Sep 17 17:38:35.185010 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.184855    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="prometheus" containerID="cri-o://a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697" gracePeriod=600
Sep 17 17:38:35.185010 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.184856    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="config-reloader" containerID="cri-o://6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e" gracePeriod=600
Sep 17 17:38:35.185268 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:35.185072    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-web" containerID="cri-o://771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07" gracePeriod=600
Sep 17 17:38:36.011770 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011740    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d" exitCode=0
Sep 17 17:38:36.011770 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011763    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83" exitCode=0
Sep 17 17:38:36.011770 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011768    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241" exitCode=0
Sep 17 17:38:36.011770 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011774    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e" exitCode=0
Sep 17 17:38:36.011770 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011780    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697" exitCode=0
Sep 17 17:38:36.012055 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d"}
Sep 17 17:38:36.012055 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83"}
Sep 17 17:38:36.012055 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241"}
Sep 17 17:38:36.012055 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e"}
Sep 17 17:38:36.012055 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.011946    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697"}
Sep 17 17:38:36.432644 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.432611    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:36.542333 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-2\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542504 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542350    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-kubelet-serving-ca-bundle\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542504 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542375    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-trusted-ca-bundle\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542504 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542422    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-metrics-client-certs\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542504 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542449    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-metrics-client-ca\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542504 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-kube-rbac-proxy\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542504    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-serving-certs-ca-bundle\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542532    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6qh9\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-kube-api-access-j6qh9\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-0\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542660    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-config-out\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-db\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542714    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-thanos-prometheus-http-client-file\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542722    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-grpc-tls\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.542794 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-1\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542812    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-tls-assets\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542887    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-tls\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542919    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-config\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542941    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542957    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-web-config\") pod \"d95c072e-b110-42ad-9977-e1b92e39f9d6\" (UID: \"d95c072e-b110-42ad-9977-e1b92e39f9d6\") "
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.542991    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.543182    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.543199    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.543213    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.543229    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-metrics-client-ca\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.543239 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.543243    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.543931 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.543772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.544302 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.544032    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:36.544818 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.544790    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:36.547306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.545308    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.547306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.546527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.547306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.546793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.547306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.546849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.547306 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.547112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.547615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.547445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-config-out" (OuterVolumeSpecName: "config-out") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:36.547615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.547498    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-kube-api-access-j6qh9" (OuterVolumeSpecName: "kube-api-access-j6qh9") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "kube-api-access-j6qh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:36.547796 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.547740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:36.547796 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.547771    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.547968 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.547949    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-config" (OuterVolumeSpecName: "config") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.548817 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.548796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.557620 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.557591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-web-config" (OuterVolumeSpecName: "web-config") pod "d95c072e-b110-42ad-9977-e1b92e39f9d6" (UID: "d95c072e-b110-42ad-9977-e1b92e39f9d6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:36.643522 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643501    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-metrics-client-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643522 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643521    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-kube-rbac-proxy\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643531    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643542    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qh9\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-kube-api-access-j6qh9\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643551    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643560    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-config-out\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643568    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-db\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643576    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643585    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-grpc-tls\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643592    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d95c072e-b110-42ad-9977-e1b92e39f9d6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643601    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d95c072e-b110-42ad-9977-e1b92e39f9d6-tls-assets\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643609    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643616    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643625    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-config\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:36.643650 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:36.643651    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d95c072e-b110-42ad-9977-e1b92e39f9d6-web-config\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:38:37.019718 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.019692    2561 generic.go:356] "Generic (PLEG): container finished" podID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerID="771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07" exitCode=0
Sep 17 17:38:37.019848 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.019823    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.019848 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.019824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07"}
Sep 17 17:38:37.019931 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.019865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d95c072e-b110-42ad-9977-e1b92e39f9d6","Type":"ContainerDied","Data":"b9cf256c14aa5d8e39c917634c9df00d6e931b381b6be3b0c7501c03aca3171f"}
Sep 17 17:38:37.019931 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.019896    2561 scope.go:117] "RemoveContainer" containerID="5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d"
Sep 17 17:38:37.033513 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.031033    2561 scope.go:117] "RemoveContainer" containerID="344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83"
Sep 17 17:38:37.042433 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.042412    2561 scope.go:117] "RemoveContainer" containerID="771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07"
Sep 17 17:38:37.049421 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.049404    2561 scope.go:117] "RemoveContainer" containerID="ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241"
Sep 17 17:38:37.055620 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.055604    2561 scope.go:117] "RemoveContainer" containerID="6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e"
Sep 17 17:38:37.061781 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.061755    2561 scope.go:117] "RemoveContainer" containerID="a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697"
Sep 17 17:38:37.063889 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.063868    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:38:37.068888 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.068870    2561 scope.go:117] "RemoveContainer" containerID="db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad"
Sep 17 17:38:37.073602 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.073580    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:38:37.075506 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.075485    2561 scope.go:117] "RemoveContainer" containerID="5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d"
Sep 17 17:38:37.075772 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.075755    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d\": container with ID starting with 5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d not found: ID does not exist" containerID="5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d"
Sep 17 17:38:37.075822 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.075778    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d"} err="failed to get container status \"5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d\": rpc error: code = NotFound desc = could not find container \"5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d\": container with ID starting with 5622ee3fc343b83157470d4e9bc7747a446070359361103043d10e6394b8890d not found: ID does not exist"
Sep 17 17:38:37.075822 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.075794    2561 scope.go:117] "RemoveContainer" containerID="344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83"
Sep 17 17:38:37.076004 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.075987    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83\": container with ID starting with 344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83 not found: ID does not exist" containerID="344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83"
Sep 17 17:38:37.076043 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076008    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83"} err="failed to get container status \"344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83\": rpc error: code = NotFound desc = could not find container \"344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83\": container with ID starting with 344ddcbeace00336fb97f37d37ecaac22c2e8ae46ddc1e3d358ca8d1def5ca83 not found: ID does not exist"
Sep 17 17:38:37.076043 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076023    2561 scope.go:117] "RemoveContainer" containerID="771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07"
Sep 17 17:38:37.076278 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.076264    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07\": container with ID starting with 771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07 not found: ID does not exist" containerID="771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07"
Sep 17 17:38:37.076328 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076279    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07"} err="failed to get container status \"771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07\": rpc error: code = NotFound desc = could not find container \"771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07\": container with ID starting with 771db28b093f75d5e7a1d6f017115d983d5804455b284a81a8e01a0daefbdf07 not found: ID does not exist"
Sep 17 17:38:37.076328 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076290    2561 scope.go:117] "RemoveContainer" containerID="ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241"
Sep 17 17:38:37.076504 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.076491    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241\": container with ID starting with ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241 not found: ID does not exist" containerID="ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241"
Sep 17 17:38:37.076548 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076506    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241"} err="failed to get container status \"ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241\": rpc error: code = NotFound desc = could not find container \"ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241\": container with ID starting with ac18a8fbd2f5eea7ee917b07ff55ab65866b0a817710077be81bad00e62fc241 not found: ID does not exist"
Sep 17 17:38:37.076548 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076517    2561 scope.go:117] "RemoveContainer" containerID="6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e"
Sep 17 17:38:37.076763 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.076746    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e\": container with ID starting with 6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e not found: ID does not exist" containerID="6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e"
Sep 17 17:38:37.076830 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076767    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e"} err="failed to get container status \"6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e\": rpc error: code = NotFound desc = could not find container \"6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e\": container with ID starting with 6341cb1043669d847903e39dc5f38307863133efa198f26437ac09b77327131e not found: ID does not exist"
Sep 17 17:38:37.076830 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.076781    2561 scope.go:117] "RemoveContainer" containerID="a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697"
Sep 17 17:38:37.077028 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.077012    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697\": container with ID starting with a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697 not found: ID does not exist" containerID="a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697"
Sep 17 17:38:37.077072 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.077031    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697"} err="failed to get container status \"a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697\": rpc error: code = NotFound desc = could not find container \"a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697\": container with ID starting with a651822fd6a449169a61d7bb01e2382a5d6e26f6b52a01a347851cfe3ca21697 not found: ID does not exist"
Sep 17 17:38:37.077072 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.077044    2561 scope.go:117] "RemoveContainer" containerID="db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad"
Sep 17 17:38:37.077243 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:38:37.077230    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad\": container with ID starting with db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad not found: ID does not exist" containerID="db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad"
Sep 17 17:38:37.077278 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.077246    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad"} err="failed to get container status \"db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad\": rpc error: code = NotFound desc = could not find container \"db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad\": container with ID starting with db7f46194b6b9a6b31b2e203be3d76a403da2d380fb68cbbde018b1a18abb0ad not found: ID does not exist"
Sep 17 17:38:37.128207 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128188    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:38:37.128489 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128470    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" containerName="registry"
Sep 17 17:38:37.128489 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128485    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" containerName="registry"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128510    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="thanos-sidecar"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="thanos-sidecar"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128524    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-thanos"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128532    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-thanos"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128541    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="prometheus"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128547    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="prometheus"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128554    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-web"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128559    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-web"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128566    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="init-config-reloader"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128571    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="init-config-reloader"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128578    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="config-reloader"
Sep 17 17:38:37.128599 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128584    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="config-reloader"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128653    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-web"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128663    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128673    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="thanos-sidecar"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128679    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="kube-rbac-proxy-thanos"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128685    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="prometheus"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128689    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" containerName="config-reloader"
Sep 17 17:38:37.128993 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.128694    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f19b9e24-40a5-4a33-8f8e-24a299fb0f02" containerName="registry"
Sep 17 17:38:37.134492 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.134475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.140000 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.139984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 17:38:37.140082 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.140054    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 17:38:37.141009 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.140989    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 17:38:37.141102 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.141077    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hlvrj\""
Sep 17 17:38:37.141196 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.141177    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 17:38:37.141308 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.141292    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 17:38:37.144375 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.144358    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 17:38:37.144484 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.144405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 17:38:37.144766 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.144750    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 17:38:37.144818 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.144768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1cirbv1237r1j\""
Sep 17 17:38:37.145591 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.145577    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 17:38:37.145753 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.145725    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 17:38:37.146030 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.146016    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 17:38:37.147056 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.147039    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 17:38:37.147998 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.147980    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 17:38:37.151082 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.151062    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 17:38:37.158079 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.158054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:38:37.247493 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247468    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-web-config\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbxb\" (UniqueName: \"kubernetes.io/projected/1155a300-f1e8-4cf6-9b79-fe4420a87288-kube-api-access-fwbxb\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1155a300-f1e8-4cf6-9b79-fe4420a87288-config-out\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1155a300-f1e8-4cf6-9b79-fe4420a87288-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247788    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.247836 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.248024 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.248024 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.248024 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.248024 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.248024 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-config\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.248024 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.247933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.348890 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.348864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.348996 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.348902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.348996 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.348925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349074 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349049    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-config\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349109 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349145 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-web-config\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349175 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349223 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349273 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbxb\" (UniqueName: \"kubernetes.io/projected/1155a300-f1e8-4cf6-9b79-fe4420a87288-kube-api-access-fwbxb\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349273 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349369 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1155a300-f1e8-4cf6-9b79-fe4420a87288-config-out\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349369 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349294    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349369 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349369 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349480    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1155a300-f1e8-4cf6-9b79-fe4420a87288-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349902 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.349955 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.349905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.350244 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.350208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.351012 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.350985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.351012 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.351006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.352626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.351504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.352626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.352162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.352626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.352373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.352626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.352507    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.352626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.352562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-config\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.352626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.352586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.353067 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.353043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1155a300-f1e8-4cf6-9b79-fe4420a87288-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.353589 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.353562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.354605 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.354581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1155a300-f1e8-4cf6-9b79-fe4420a87288-config-out\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.354713 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.354591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.354797 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.354782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-web-config\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.354987 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.354965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1155a300-f1e8-4cf6-9b79-fe4420a87288-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.355507 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.355490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.356004 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.355982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1155a300-f1e8-4cf6-9b79-fe4420a87288-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.359909 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.359888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbxb\" (UniqueName: \"kubernetes.io/projected/1155a300-f1e8-4cf6-9b79-fe4420a87288-kube-api-access-fwbxb\") pod \"prometheus-k8s-0\" (UID: \"1155a300-f1e8-4cf6-9b79-fe4420a87288\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.444053 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.444032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:38:37.757520 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:37.757493    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 17:38:37.759325 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:38:37.759302    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1155a300_f1e8_4cf6_9b79_fe4420a87288.slice/crio-b8653f697c5e64d2b2be26c9df4f327282ec44c4010ae8de4e5a249f6b41f7ee WatchSource:0}: Error finding container b8653f697c5e64d2b2be26c9df4f327282ec44c4010ae8de4e5a249f6b41f7ee: Status 404 returned error can't find the container with id b8653f697c5e64d2b2be26c9df4f327282ec44c4010ae8de4e5a249f6b41f7ee
Sep 17 17:38:38.025049 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:38.024973    2561 generic.go:356] "Generic (PLEG): container finished" podID="1155a300-f1e8-4cf6-9b79-fe4420a87288" containerID="2fd48df67b91b1cba1e30f466d3d1d29bef60c8c114cad4cfe9aa1a7dddc758a" exitCode=0
Sep 17 17:38:38.025188 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:38.025063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerDied","Data":"2fd48df67b91b1cba1e30f466d3d1d29bef60c8c114cad4cfe9aa1a7dddc758a"}
Sep 17 17:38:38.025188 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:38.025142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"b8653f697c5e64d2b2be26c9df4f327282ec44c4010ae8de4e5a249f6b41f7ee"}
Sep 17 17:38:39.019569 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.019540    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95c072e-b110-42ad-9977-e1b92e39f9d6" path="/var/lib/kubelet/pods/d95c072e-b110-42ad-9977-e1b92e39f9d6/volumes"
Sep 17 17:38:39.032668 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.032623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"fe1ce6a474aa8d9cdb0170212335e999ffbe922b485307ee67c458b175474ed5"}
Sep 17 17:38:39.032789 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.032696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"7d8a178c19ad92a62f5fd34d0ac7ae5624192b3de55d9f700ebcd4943b15eabf"}
Sep 17 17:38:39.032789 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.032722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"99d53ca876d80b2d7869855bb972f1c7e84e40b7b924eb26968b9efaa6c196b1"}
Sep 17 17:38:39.032789 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.032744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"0dcff95da0fe2d9289a5e2fd7e5fea6e2455fbf633490bb8a9ffeb82d0ee9040"}
Sep 17 17:38:39.032789 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.032769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"203bea610a9054934853988235a0d53cead25d6a11b468e1302e5aa0d10a9e37"}
Sep 17 17:38:39.032916 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.032793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1155a300-f1e8-4cf6-9b79-fe4420a87288","Type":"ContainerStarted","Data":"e4d34e6a0930482b783f3c2f00753d3e1fd9e47b9bf7335be9976294daae4975"}
Sep 17 17:38:39.069414 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:39.069366    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.048923254 podStartE2EDuration="2.069351782s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.428528ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:39.063705115 +0000 UTC m=+244.023590385" watchObservedRunningTime="2026-09-17 17:38:39.069351782 +0000 UTC m=+244.029237047"
Sep 17 17:38:42.444806 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:38:42.444768    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:36.934860 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:39:36.934834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:39:36.938829 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:39:36.938805    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:39:36.944756 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:39:36.944734    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:39:37.444819 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:39:37.444774    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:37.460052 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:39:37.460027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:39:38.271469 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:39:38.271441    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 17:44:36.960259 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:44:36.960231    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:44:36.963117 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:44:36.963095    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:49:36.985242 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:49:36.985208    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:49:36.985711 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:49:36.985591    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:50:39.318784 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.318748    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"]
Sep 17 17:50:39.321777 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.321762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.325521 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.325493    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 17:50:39.325783 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.325767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 17:50:39.326775 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.326762    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-42msp\""
Sep 17 17:50:39.336600 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.336570    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"]
Sep 17 17:50:39.407429 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.407399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1994b710-8dbc-49d2-91c3-bae01b47fa59-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vfmmv\" (UID: \"1994b710-8dbc-49d2-91c3-bae01b47fa59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.407429 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.407433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2g4\" (UniqueName: \"kubernetes.io/projected/1994b710-8dbc-49d2-91c3-bae01b47fa59-kube-api-access-tt2g4\") pod \"cert-manager-cainjector-559cdd8c87-vfmmv\" (UID: \"1994b710-8dbc-49d2-91c3-bae01b47fa59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.507974 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.507942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1994b710-8dbc-49d2-91c3-bae01b47fa59-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vfmmv\" (UID: \"1994b710-8dbc-49d2-91c3-bae01b47fa59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.507974 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.507978    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2g4\" (UniqueName: \"kubernetes.io/projected/1994b710-8dbc-49d2-91c3-bae01b47fa59-kube-api-access-tt2g4\") pod \"cert-manager-cainjector-559cdd8c87-vfmmv\" (UID: \"1994b710-8dbc-49d2-91c3-bae01b47fa59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.517122 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.517092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1994b710-8dbc-49d2-91c3-bae01b47fa59-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vfmmv\" (UID: \"1994b710-8dbc-49d2-91c3-bae01b47fa59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.517226 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.517215    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2g4\" (UniqueName: \"kubernetes.io/projected/1994b710-8dbc-49d2-91c3-bae01b47fa59-kube-api-access-tt2g4\") pod \"cert-manager-cainjector-559cdd8c87-vfmmv\" (UID: \"1994b710-8dbc-49d2-91c3-bae01b47fa59\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.639698 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.639615    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"
Sep 17 17:50:39.908125 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.908046    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv"]
Sep 17 17:50:39.911171 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:50:39.911144    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1994b710_8dbc_49d2_91c3_bae01b47fa59.slice/crio-80401f4aed74b39fc0c04958d22ec2f76a883917b7dc5fe4b9a36aafafad7398 WatchSource:0}: Error finding container 80401f4aed74b39fc0c04958d22ec2f76a883917b7dc5fe4b9a36aafafad7398: Status 404 returned error can't find the container with id 80401f4aed74b39fc0c04958d22ec2f76a883917b7dc5fe4b9a36aafafad7398
Sep 17 17:50:39.913023 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:39.913006    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:50:40.703909 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:40.703875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv" event={"ID":"1994b710-8dbc-49d2-91c3-bae01b47fa59","Type":"ContainerStarted","Data":"80401f4aed74b39fc0c04958d22ec2f76a883917b7dc5fe4b9a36aafafad7398"}
Sep 17 17:50:44.721179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:44.721147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv" event={"ID":"1994b710-8dbc-49d2-91c3-bae01b47fa59","Type":"ContainerStarted","Data":"9e993b166bfbcc6e88ee5c4918622b4b8d815ea45308a11af62f5015b577293b"}
Sep 17 17:50:44.743245 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:44.743200    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vfmmv" podStartSLOduration=1.86283632 podStartE2EDuration="5.743184953s" totalImagesPullingTime="3.880348633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:44.741414187 +0000 UTC m=+969.701299462" watchObservedRunningTime="2026-09-17 17:50:44.743184953 +0000 UTC m=+969.703070222"
Sep 17 17:50:51.126092 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.126020    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"]
Sep 17 17:50:51.129437 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.129423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.132131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.132112    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:50:51.132131 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.132119    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 17:50:51.132978 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.132963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-msgfr\""
Sep 17 17:50:51.139047 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.139025    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"]
Sep 17 17:50:51.207541 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.207513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90930738-b7d8-42a2-8c32-eccd4a85be80-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nk4ds\" (UID: \"90930738-b7d8-42a2-8c32-eccd4a85be80\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.207541 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.207545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdb5\" (UniqueName: \"kubernetes.io/projected/90930738-b7d8-42a2-8c32-eccd4a85be80-kube-api-access-2pdb5\") pod \"openshift-lws-operator-fd8ccff4c-nk4ds\" (UID: \"90930738-b7d8-42a2-8c32-eccd4a85be80\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.308102 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.308077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb5\" (UniqueName: \"kubernetes.io/projected/90930738-b7d8-42a2-8c32-eccd4a85be80-kube-api-access-2pdb5\") pod \"openshift-lws-operator-fd8ccff4c-nk4ds\" (UID: \"90930738-b7d8-42a2-8c32-eccd4a85be80\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.308199 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.308145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90930738-b7d8-42a2-8c32-eccd4a85be80-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nk4ds\" (UID: \"90930738-b7d8-42a2-8c32-eccd4a85be80\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.308461 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.308446    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90930738-b7d8-42a2-8c32-eccd4a85be80-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nk4ds\" (UID: \"90930738-b7d8-42a2-8c32-eccd4a85be80\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.317275 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.317246    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdb5\" (UniqueName: \"kubernetes.io/projected/90930738-b7d8-42a2-8c32-eccd4a85be80-kube-api-access-2pdb5\") pod \"openshift-lws-operator-fd8ccff4c-nk4ds\" (UID: \"90930738-b7d8-42a2-8c32-eccd4a85be80\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.438534 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.438456    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"
Sep 17 17:50:51.711355 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.711275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds"]
Sep 17 17:50:51.714277 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:50:51.714250    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90930738_b7d8_42a2_8c32_eccd4a85be80.slice/crio-0f998a2921d294eb54dc34774cb89a2a16043c0d4fe9df98af1af2740f6b7721 WatchSource:0}: Error finding container 0f998a2921d294eb54dc34774cb89a2a16043c0d4fe9df98af1af2740f6b7721: Status 404 returned error can't find the container with id 0f998a2921d294eb54dc34774cb89a2a16043c0d4fe9df98af1af2740f6b7721
Sep 17 17:50:51.748917 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:51.748882    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds" event={"ID":"90930738-b7d8-42a2-8c32-eccd4a85be80","Type":"ContainerStarted","Data":"0f998a2921d294eb54dc34774cb89a2a16043c0d4fe9df98af1af2740f6b7721"}
Sep 17 17:50:54.763908 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:54.763874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds" event={"ID":"90930738-b7d8-42a2-8c32-eccd4a85be80","Type":"ContainerStarted","Data":"1bd1dca742f7c982b4f1a3709331a23990af1563a6967e12f63352a2860becbd"}
Sep 17 17:50:54.781845 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:50:54.781792    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nk4ds" podStartSLOduration=1.254013777 podStartE2EDuration="3.781773601s" totalImagesPullingTime="2.527759824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:54.781525177 +0000 UTC m=+979.741410462" watchObservedRunningTime="2026-09-17 17:50:54.781773601 +0000 UTC m=+979.741658873"
Sep 17 17:51:06.352824 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.352787    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"]
Sep 17 17:51:06.357580 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.357560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.361094 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.361071    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 17:51:06.361094 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.361086    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tdsdt\""
Sep 17 17:51:06.361236 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.361109    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 17:51:06.361236 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.361182    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 17:51:06.370310 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.370289    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"]
Sep 17 17:51:06.528449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.528415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xghv\" (UniqueName: \"kubernetes.io/projected/dac92b04-4493-4d46-9ae3-48550b88a6fb-kube-api-access-8xghv\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.528611 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.528455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dac92b04-4493-4d46-9ae3-48550b88a6fb-metrics-cert\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.528611 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.528524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dac92b04-4493-4d46-9ae3-48550b88a6fb-cert\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.528611 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.528563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dac92b04-4493-4d46-9ae3-48550b88a6fb-manager-config\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.629778 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.629702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dac92b04-4493-4d46-9ae3-48550b88a6fb-manager-config\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.629778 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.629752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xghv\" (UniqueName: \"kubernetes.io/projected/dac92b04-4493-4d46-9ae3-48550b88a6fb-kube-api-access-8xghv\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.629778 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.629776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dac92b04-4493-4d46-9ae3-48550b88a6fb-metrics-cert\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.629991 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.629820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dac92b04-4493-4d46-9ae3-48550b88a6fb-cert\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.630374 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.630350    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dac92b04-4493-4d46-9ae3-48550b88a6fb-manager-config\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.632137 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.632110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dac92b04-4493-4d46-9ae3-48550b88a6fb-metrics-cert\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.632237 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.632221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dac92b04-4493-4d46-9ae3-48550b88a6fb-cert\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.644098 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.644073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xghv\" (UniqueName: \"kubernetes.io/projected/dac92b04-4493-4d46-9ae3-48550b88a6fb-kube-api-access-8xghv\") pod \"lws-controller-manager-6796bd7874-6hlb9\" (UID: \"dac92b04-4493-4d46-9ae3-48550b88a6fb\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.667065 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.667043    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:06.949352 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:06.949262    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"]
Sep 17 17:51:06.952731 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:51:06.952702    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac92b04_4493_4d46_9ae3_48550b88a6fb.slice/crio-d98e2b9046b6f14f2fdb7c3afa8299ace0478d2acd8be29924b6c4cf15bb0b14 WatchSource:0}: Error finding container d98e2b9046b6f14f2fdb7c3afa8299ace0478d2acd8be29924b6c4cf15bb0b14: Status 404 returned error can't find the container with id d98e2b9046b6f14f2fdb7c3afa8299ace0478d2acd8be29924b6c4cf15bb0b14
Sep 17 17:51:07.825274 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:07.825234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9" event={"ID":"dac92b04-4493-4d46-9ae3-48550b88a6fb","Type":"ContainerStarted","Data":"d98e2b9046b6f14f2fdb7c3afa8299ace0478d2acd8be29924b6c4cf15bb0b14"}
Sep 17 17:51:08.830910 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:08.830875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9" event={"ID":"dac92b04-4493-4d46-9ae3-48550b88a6fb","Type":"ContainerStarted","Data":"1d19fda5ca3fbb14cf527cd168c0399670f4aeb8c139b57dff13bbc8e347a757"}
Sep 17 17:51:08.831274 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:08.830968    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:08.851795 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:08.851744    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9" podStartSLOduration=1.182623713 podStartE2EDuration="2.851725573s" totalImagesPullingTime="1.66910186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:08.851105776 +0000 UTC m=+993.810991047" watchObservedRunningTime="2026-09-17 17:51:08.851725573 +0000 UTC m=+993.811610844"
Sep 17 17:51:15.608435 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.608406    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"]
Sep 17 17:51:15.615508 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.615492    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.619171 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.619149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:51:15.619517 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.619499    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:51:15.619623 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.619549    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:51:15.619623 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.619618    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-zwkch\""
Sep 17 17:51:15.631357 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.631335    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"]
Sep 17 17:51:15.705316 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705492 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrvs\" (UniqueName: \"kubernetes.io/projected/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-kube-api-access-pbrvs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705492 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705492 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705492 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705721 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705721 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705721 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.705721 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.705623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806358 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806358 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806604 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806604 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806604 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806604 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806604 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrvs\" (UniqueName: \"kubernetes.io/projected/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-kube-api-access-pbrvs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806604 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806585    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806988 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806988 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.806988 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.807086 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.806995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.807182 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.807155    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.807315 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.807185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.808759 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.808735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.808968 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.808952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.834896 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.834877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.846815 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.846792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrvs\" (UniqueName: \"kubernetes.io/projected/8bf9b8f0-b273-4f7a-8ee4-be8df6445e18-kube-api-access-pbrvs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j\" (UID: \"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:15.926800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:15.926730    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:16.203008 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:16.202937    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"]
Sep 17 17:51:16.205853 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:51:16.205826    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf9b8f0_b273_4f7a_8ee4_be8df6445e18.slice/crio-475c08571a6d8547ce7d9153eeb50e0a01c1e1defce93ef10e4dce9e2783ced1 WatchSource:0}: Error finding container 475c08571a6d8547ce7d9153eeb50e0a01c1e1defce93ef10e4dce9e2783ced1: Status 404 returned error can't find the container with id 475c08571a6d8547ce7d9153eeb50e0a01c1e1defce93ef10e4dce9e2783ced1
Sep 17 17:51:16.862694 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:16.862661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" event={"ID":"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18","Type":"ContainerStarted","Data":"475c08571a6d8547ce7d9153eeb50e0a01c1e1defce93ef10e4dce9e2783ced1"}
Sep 17 17:51:19.837314 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:19.837279    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-6hlb9"
Sep 17 17:51:21.305923 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:21.305887    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:51:21.306218 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:21.305956    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:51:21.306218 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:21.305985    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:51:21.885886 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:21.885853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" event={"ID":"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18","Type":"ContainerStarted","Data":"9b74c5235c234addb82f52f07864b96930ad5753782564b3b96eb856e2eb7bc2"}
Sep 17 17:51:21.910644 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:21.910573    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" podStartSLOduration=1.812654587 podStartE2EDuration="6.910556993s" totalImagesPullingTime="5.097902406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:21.909079271 +0000 UTC m=+1006.868964541" watchObservedRunningTime="2026-09-17 17:51:21.910556993 +0000 UTC m=+1006.870442262"
Sep 17 17:51:21.927712 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:21.927688    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:22.931847 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:22.931819    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:23.894286 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:23.894255    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:51:23.895073 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:51:23.895058    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:52:00.915330 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:00.915298    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"]
Sep 17 17:52:00.923498 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:00.923478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:00.926579 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:00.926547    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:52:00.926714 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:00.926605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ccw85\""
Sep 17 17:52:00.927569 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:00.927552    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:52:00.951658 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:00.949647    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"]
Sep 17 17:52:01.072025 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:01.071992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rmt\" (UniqueName: \"kubernetes.io/projected/28ff21c9-b685-4e40-bf3c-5ca76fc32334-kube-api-access-h2rmt\") pod \"authorino-operator-6fc8f864f4-zkvgz\" (UID: \"28ff21c9-b685-4e40-bf3c-5ca76fc32334\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:01.173236 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:01.173161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rmt\" (UniqueName: \"kubernetes.io/projected/28ff21c9-b685-4e40-bf3c-5ca76fc32334-kube-api-access-h2rmt\") pod \"authorino-operator-6fc8f864f4-zkvgz\" (UID: \"28ff21c9-b685-4e40-bf3c-5ca76fc32334\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:01.185085 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:01.185051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rmt\" (UniqueName: \"kubernetes.io/projected/28ff21c9-b685-4e40-bf3c-5ca76fc32334-kube-api-access-h2rmt\") pod \"authorino-operator-6fc8f864f4-zkvgz\" (UID: \"28ff21c9-b685-4e40-bf3c-5ca76fc32334\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:01.233252 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:01.233224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:01.531397 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:01.531371    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"]
Sep 17 17:52:01.533516 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:52:01.533486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ff21c9_b685_4e40_bf3c_5ca76fc32334.slice/crio-b1fe06378101b7c91de20d15dfca95c0f1ddfdfdd8d80c45464fa64a2b11798b WatchSource:0}: Error finding container b1fe06378101b7c91de20d15dfca95c0f1ddfdfdd8d80c45464fa64a2b11798b: Status 404 returned error can't find the container with id b1fe06378101b7c91de20d15dfca95c0f1ddfdfdd8d80c45464fa64a2b11798b
Sep 17 17:52:02.037607 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:02.037574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz" event={"ID":"28ff21c9-b685-4e40-bf3c-5ca76fc32334","Type":"ContainerStarted","Data":"b1fe06378101b7c91de20d15dfca95c0f1ddfdfdd8d80c45464fa64a2b11798b"}
Sep 17 17:52:05.054251 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:05.054220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz" event={"ID":"28ff21c9-b685-4e40-bf3c-5ca76fc32334","Type":"ContainerStarted","Data":"5506ba87554ee64fd51df80c7b8faf9ac303039ed83b1de4c072ba7ec1aca22f"}
Sep 17 17:52:05.054665 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:05.054353    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:16.060327 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:16.060296    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz"
Sep 17 17:52:16.089828 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:16.089777    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-zkvgz" podStartSLOduration=13.197481347 podStartE2EDuration="16.089759616s" totalImagesPullingTime="2.892278269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:05.076246587 +0000 UTC m=+1050.036131857" watchObservedRunningTime="2026-09-17 17:52:16.089759616 +0000 UTC m=+1061.049644877"
Sep 17 17:52:48.066774 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.066738    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-tshzd"]
Sep 17 17:52:48.074754 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.074731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:52:48.078134 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.078118    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zkfpq\""
Sep 17 17:52:48.091281 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.091260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tshzd"]
Sep 17 17:52:48.173386 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.173354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxm2\" (UniqueName: \"kubernetes.io/projected/b5518f16-e4e3-4283-976e-0c2fdeb796c6-kube-api-access-pgxm2\") pod \"authorino-59c974bd48-tshzd\" (UID: \"b5518f16-e4e3-4283-976e-0c2fdeb796c6\") " pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:52:48.274192 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.274165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxm2\" (UniqueName: \"kubernetes.io/projected/b5518f16-e4e3-4283-976e-0c2fdeb796c6-kube-api-access-pgxm2\") pod \"authorino-59c974bd48-tshzd\" (UID: \"b5518f16-e4e3-4283-976e-0c2fdeb796c6\") " pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:52:48.284251 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.284233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxm2\" (UniqueName: \"kubernetes.io/projected/b5518f16-e4e3-4283-976e-0c2fdeb796c6-kube-api-access-pgxm2\") pod \"authorino-59c974bd48-tshzd\" (UID: \"b5518f16-e4e3-4283-976e-0c2fdeb796c6\") " pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:52:48.383602 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.383522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:52:48.659939 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:48.659911    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tshzd"]
Sep 17 17:52:48.662008 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:52:48.661964    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5518f16_e4e3_4283_976e_0c2fdeb796c6.slice/crio-07a2e28436d366b1841f9a9980d863a33bd253b428e93bd235174c1f224fd0b7 WatchSource:0}: Error finding container 07a2e28436d366b1841f9a9980d863a33bd253b428e93bd235174c1f224fd0b7: Status 404 returned error can't find the container with id 07a2e28436d366b1841f9a9980d863a33bd253b428e93bd235174c1f224fd0b7
Sep 17 17:52:49.246626 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:49.246593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tshzd" event={"ID":"b5518f16-e4e3-4283-976e-0c2fdeb796c6","Type":"ContainerStarted","Data":"07a2e28436d366b1841f9a9980d863a33bd253b428e93bd235174c1f224fd0b7"}
Sep 17 17:52:51.256398 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:51.256362    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tshzd" event={"ID":"b5518f16-e4e3-4283-976e-0c2fdeb796c6","Type":"ContainerStarted","Data":"63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22"}
Sep 17 17:52:51.276174 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:52:51.276082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-tshzd" podStartSLOduration=0.92524451 podStartE2EDuration="3.276066374s" totalImagesPullingTime="2.350821864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:51.274946659 +0000 UTC m=+1096.234831930" watchObservedRunningTime="2026-09-17 17:52:51.276066374 +0000 UTC m=+1096.235951643"
Sep 17 17:53:11.289308 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.289266    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5zndp"]
Sep 17 17:53:11.293762 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.293735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.296559 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.296538    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 17:53:11.301367 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.301347    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5zndp"]
Sep 17 17:53:11.475728 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.475688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jnwm\" (UniqueName: \"kubernetes.io/projected/16f07158-3254-45a8-8cee-e2637109b2b6-kube-api-access-8jnwm\") pod \"authorino-6d9898bd99-5zndp\" (UID: \"16f07158-3254-45a8-8cee-e2637109b2b6\") " pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.475907 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.475760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f07158-3254-45a8-8cee-e2637109b2b6-tls-cert\") pod \"authorino-6d9898bd99-5zndp\" (UID: \"16f07158-3254-45a8-8cee-e2637109b2b6\") " pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.576545 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.576445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f07158-3254-45a8-8cee-e2637109b2b6-tls-cert\") pod \"authorino-6d9898bd99-5zndp\" (UID: \"16f07158-3254-45a8-8cee-e2637109b2b6\") " pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.576545 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.576537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jnwm\" (UniqueName: \"kubernetes.io/projected/16f07158-3254-45a8-8cee-e2637109b2b6-kube-api-access-8jnwm\") pod \"authorino-6d9898bd99-5zndp\" (UID: \"16f07158-3254-45a8-8cee-e2637109b2b6\") " pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.579057 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.579034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16f07158-3254-45a8-8cee-e2637109b2b6-tls-cert\") pod \"authorino-6d9898bd99-5zndp\" (UID: \"16f07158-3254-45a8-8cee-e2637109b2b6\") " pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.585195 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.585173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jnwm\" (UniqueName: \"kubernetes.io/projected/16f07158-3254-45a8-8cee-e2637109b2b6-kube-api-access-8jnwm\") pod \"authorino-6d9898bd99-5zndp\" (UID: \"16f07158-3254-45a8-8cee-e2637109b2b6\") " pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.603124 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.603106    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5zndp"
Sep 17 17:53:11.883332 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:11.883253    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5zndp"]
Sep 17 17:53:11.886023 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:53:11.885995    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16f07158_3254_45a8_8cee_e2637109b2b6.slice/crio-57cb682427280d00b00d2d35505e72fdeac7fd18e4902d1d2904d4c9606321d4 WatchSource:0}: Error finding container 57cb682427280d00b00d2d35505e72fdeac7fd18e4902d1d2904d4c9606321d4: Status 404 returned error can't find the container with id 57cb682427280d00b00d2d35505e72fdeac7fd18e4902d1d2904d4c9606321d4
Sep 17 17:53:12.348826 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:12.348793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5zndp" event={"ID":"16f07158-3254-45a8-8cee-e2637109b2b6","Type":"ContainerStarted","Data":"57cb682427280d00b00d2d35505e72fdeac7fd18e4902d1d2904d4c9606321d4"}
Sep 17 17:53:13.354792 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.354755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5zndp" event={"ID":"16f07158-3254-45a8-8cee-e2637109b2b6","Type":"ContainerStarted","Data":"ef8bffebc20eed52d5c22e8deed84c1cd608937a6914a503797b435ec85ef7bc"}
Sep 17 17:53:13.375523 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.375476    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-5zndp" podStartSLOduration=1.745905414 podStartE2EDuration="2.375458129s" totalImagesPullingTime="629.552715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:13.374085067 +0000 UTC m=+1118.333970336" watchObservedRunningTime="2026-09-17 17:53:13.375458129 +0000 UTC m=+1118.335343428"
Sep 17 17:53:13.404350 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.404320    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tshzd"]
Sep 17 17:53:13.404545 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.404521    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-tshzd" podUID="b5518f16-e4e3-4283-976e-0c2fdeb796c6" containerName="authorino" containerID="cri-o://63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22" gracePeriod=30
Sep 17 17:53:13.654207 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.654184    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:53:13.692476 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.692450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgxm2\" (UniqueName: \"kubernetes.io/projected/b5518f16-e4e3-4283-976e-0c2fdeb796c6-kube-api-access-pgxm2\") pod \"b5518f16-e4e3-4283-976e-0c2fdeb796c6\" (UID: \"b5518f16-e4e3-4283-976e-0c2fdeb796c6\") "
Sep 17 17:53:13.694739 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.694709    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5518f16-e4e3-4283-976e-0c2fdeb796c6-kube-api-access-pgxm2" (OuterVolumeSpecName: "kube-api-access-pgxm2") pod "b5518f16-e4e3-4283-976e-0c2fdeb796c6" (UID: "b5518f16-e4e3-4283-976e-0c2fdeb796c6"). InnerVolumeSpecName "kube-api-access-pgxm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:13.793058 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:13.793028    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxm2\" (UniqueName: \"kubernetes.io/projected/b5518f16-e4e3-4283-976e-0c2fdeb796c6-kube-api-access-pgxm2\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:53:14.360058 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.360020    2561 generic.go:356] "Generic (PLEG): container finished" podID="b5518f16-e4e3-4283-976e-0c2fdeb796c6" containerID="63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22" exitCode=0
Sep 17 17:53:14.360464 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.360083    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-tshzd"
Sep 17 17:53:14.360464 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.360102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tshzd" event={"ID":"b5518f16-e4e3-4283-976e-0c2fdeb796c6","Type":"ContainerDied","Data":"63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22"}
Sep 17 17:53:14.360464 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.360142    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-tshzd" event={"ID":"b5518f16-e4e3-4283-976e-0c2fdeb796c6","Type":"ContainerDied","Data":"07a2e28436d366b1841f9a9980d863a33bd253b428e93bd235174c1f224fd0b7"}
Sep 17 17:53:14.360464 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.360160    2561 scope.go:117] "RemoveContainer" containerID="63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22"
Sep 17 17:53:14.368184 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.368166    2561 scope.go:117] "RemoveContainer" containerID="63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22"
Sep 17 17:53:14.368445 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:53:14.368427    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22\": container with ID starting with 63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22 not found: ID does not exist" containerID="63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22"
Sep 17 17:53:14.368483 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.368454    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22"} err="failed to get container status \"63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22\": rpc error: code = NotFound desc = could not find container \"63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22\": container with ID starting with 63fd60dd2fa93521c13783d01b3505b04113964fd5f5512c37c1c4a1e19b9b22 not found: ID does not exist"
Sep 17 17:53:14.382777 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.382725    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tshzd"]
Sep 17 17:53:14.384785 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:14.384765    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-tshzd"]
Sep 17 17:53:15.022378 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:53:15.022343    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5518f16-e4e3-4283-976e-0c2fdeb796c6" path="/var/lib/kubelet/pods/b5518f16-e4e3-4283-976e-0c2fdeb796c6/volumes"
Sep 17 17:54:37.011678 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:54:37.011652    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:54:37.013199 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:54:37.013178    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:55:06.549282 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.549245    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-vjldm"]
Sep 17 17:55:06.549744 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.549625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5518f16-e4e3-4283-976e-0c2fdeb796c6" containerName="authorino"
Sep 17 17:55:06.549744 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.549657    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5518f16-e4e3-4283-976e-0c2fdeb796c6" containerName="authorino"
Sep 17 17:55:06.549744 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.549708    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5518f16-e4e3-4283-976e-0c2fdeb796c6" containerName="authorino"
Sep 17 17:55:06.552507 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.552490    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:06.555620 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.555596    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:55:06.555864 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.555844    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:55:06.555972 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.555849    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-r2jmc\""
Sep 17 17:55:06.556758 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.556744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:55:06.566164 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.566142    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vjldm"]
Sep 17 17:55:06.624482 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.624456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6d4\" (UniqueName: \"kubernetes.io/projected/08d8a2cf-48f7-4919-8c75-1f688fa92368-kube-api-access-rf6d4\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:06.624615 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.624494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a2cf-48f7-4919-8c75-1f688fa92368-tls-certs\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:06.725218 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.725133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6d4\" (UniqueName: \"kubernetes.io/projected/08d8a2cf-48f7-4919-8c75-1f688fa92368-kube-api-access-rf6d4\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:06.725364 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.725261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a2cf-48f7-4919-8c75-1f688fa92368-tls-certs\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:06.725413 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:55:06.725398    2561 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 17:55:06.725479 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:55:06.725468    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08d8a2cf-48f7-4919-8c75-1f688fa92368-tls-certs podName:08d8a2cf-48f7-4919-8c75-1f688fa92368 nodeName:}" failed. No retries permitted until 2026-09-17 17:55:07.225448921 +0000 UTC m=+1232.185334185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/08d8a2cf-48f7-4919-8c75-1f688fa92368-tls-certs") pod "model-serving-api-7b954ffc95-vjldm" (UID: "08d8a2cf-48f7-4919-8c75-1f688fa92368") : secret "model-serving-api-tls" not found
Sep 17 17:55:06.737558 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:06.737536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6d4\" (UniqueName: \"kubernetes.io/projected/08d8a2cf-48f7-4919-8c75-1f688fa92368-kube-api-access-rf6d4\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:07.229501 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:07.229469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a2cf-48f7-4919-8c75-1f688fa92368-tls-certs\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:07.231875 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:07.231857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08d8a2cf-48f7-4919-8c75-1f688fa92368-tls-certs\") pod \"model-serving-api-7b954ffc95-vjldm\" (UID: \"08d8a2cf-48f7-4919-8c75-1f688fa92368\") " pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:07.463419 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:07.463389    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:07.734986 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:07.734963    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vjldm"]
Sep 17 17:55:07.737296 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:55:07.737269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d8a2cf_48f7_4919_8c75_1f688fa92368.slice/crio-7b480d235878d7c577abc4ee4ff6050a1105c574499edac179dccbfabefddc06 WatchSource:0}: Error finding container 7b480d235878d7c577abc4ee4ff6050a1105c574499edac179dccbfabefddc06: Status 404 returned error can't find the container with id 7b480d235878d7c577abc4ee4ff6050a1105c574499edac179dccbfabefddc06
Sep 17 17:55:07.847394 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:07.847314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vjldm" event={"ID":"08d8a2cf-48f7-4919-8c75-1f688fa92368","Type":"ContainerStarted","Data":"7b480d235878d7c577abc4ee4ff6050a1105c574499edac179dccbfabefddc06"}
Sep 17 17:55:10.861730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:10.861696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vjldm" event={"ID":"08d8a2cf-48f7-4919-8c75-1f688fa92368","Type":"ContainerStarted","Data":"39c5dd4b7a1c22602c4aed74f833a41011ec431b8db1fe8a556f207be46e45da"}
Sep 17 17:55:10.862102 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:10.861819    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:55:10.882767 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:10.882720    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-vjldm" podStartSLOduration=2.809469565 podStartE2EDuration="4.882703032s" totalImagesPullingTime="2.073233467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:10.881407633 +0000 UTC m=+1235.841292908" watchObservedRunningTime="2026-09-17 17:55:10.882703032 +0000 UTC m=+1235.842588302"
Sep 17 17:55:21.870067 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:55:21.869989    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-vjldm"
Sep 17 17:57:06.442790 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.442753    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"]
Sep 17 17:57:06.446229 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.446206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.450898 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.450879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:57:06.451527 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.451505    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:57:06.451778 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.451641    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-h64nn\""
Sep 17 17:57:06.451778 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.451669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:57:06.466514 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.466482    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"]
Sep 17 17:57:06.497928 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.497891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b689574-0329-4705-81c2-35741e76ae82-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.498061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.497940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqwm\" (UniqueName: \"kubernetes.io/projected/1b689574-0329-4705-81c2-35741e76ae82-kube-api-access-9zqwm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.498061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.497986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.498061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.498002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.498061 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.498028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.498261 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.498135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599384 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599533 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599533 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b689574-0329-4705-81c2-35741e76ae82-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599533 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqwm\" (UniqueName: \"kubernetes.io/projected/1b689574-0329-4705-81c2-35741e76ae82-kube-api-access-9zqwm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599728 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599728 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599833 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599887 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.599975 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.599952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.601840 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.601812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.602076 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.602058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b689574-0329-4705-81c2-35741e76ae82-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.608349 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.608324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqwm\" (UniqueName: \"kubernetes.io/projected/1b689574-0329-4705-81c2-35741e76ae82-kube-api-access-9zqwm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:06.756580 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:06.756555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:07.032718 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:07.032696    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"]
Sep 17 17:57:07.037720 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:07.037698    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:57:07.378240 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:07.378161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" event={"ID":"1b689574-0329-4705-81c2-35741e76ae82","Type":"ContainerStarted","Data":"4267a8a249b5f192840d9bd5086f26451a5ce7f0f2f83acd094f40bb60b4ad88"}
Sep 17 17:57:09.389670 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:09.389605    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" event={"ID":"1b689574-0329-4705-81c2-35741e76ae82","Type":"ContainerStarted","Data":"4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"}
Sep 17 17:57:09.419646 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:09.419581    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" podStartSLOduration=1.565357428 podStartE2EDuration="3.419566906s" totalImagesPullingTime="1.854209478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:09.418549027 +0000 UTC m=+1354.378434298" watchObservedRunningTime="2026-09-17 17:57:09.419566906 +0000 UTC m=+1354.379452188"
Sep 17 17:57:09.757525 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:09.757489    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:16.415516 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.415429    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"]
Sep 17 17:57:16.418808 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.418784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.421738 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.421712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 17:57:16.421878 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.421767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 17:57:16.422855 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.422834    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-2z576\""
Sep 17 17:57:16.422855 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.422842    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-5zv78\""
Sep 17 17:57:16.423060 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.422971    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 17:57:16.425598 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.425579    2561 generic.go:356] "Generic (PLEG): container finished" podID="8bf9b8f0-b273-4f7a-8ee4-be8df6445e18" containerID="9b74c5235c234addb82f52f07864b96930ad5753782564b3b96eb856e2eb7bc2" exitCode=0
Sep 17 17:57:16.425718 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.425666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" event={"ID":"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18","Type":"ContainerDied","Data":"9b74c5235c234addb82f52f07864b96930ad5753782564b3b96eb856e2eb7bc2"}
Sep 17 17:57:16.426026 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.426012    2561 scope.go:117] "RemoveContainer" containerID="9b74c5235c234addb82f52f07864b96930ad5753782564b3b96eb856e2eb7bc2"
Sep 17 17:57:16.427310 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.427264    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:57:16.427427 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.427355    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:57:16.427427 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.427405    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 17:57:16.429379 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.429260    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"]
Sep 17 17:57:16.491053 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.491027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9116a93e-a372-46b2-8070-847defb20fe6-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.491188 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.491072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2l2\" (UniqueName: \"kubernetes.io/projected/9116a93e-a372-46b2-8070-847defb20fe6-kube-api-access-nz2l2\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.591507 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.591468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9116a93e-a372-46b2-8070-847defb20fe6-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.591507 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.591499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2l2\" (UniqueName: \"kubernetes.io/projected/9116a93e-a372-46b2-8070-847defb20fe6-kube-api-access-nz2l2\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.594170 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.594140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9116a93e-a372-46b2-8070-847defb20fe6-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.606776 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.606743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2l2\" (UniqueName: \"kubernetes.io/projected/9116a93e-a372-46b2-8070-847defb20fe6-kube-api-access-nz2l2\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.730996 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.730955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:16.757251 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.757220    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:16.770921 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:16.770892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:17.032080 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.031944    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"]
Sep 17 17:57:17.034535 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:57:17.034512    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9116a93e_a372_46b2_8070_847defb20fe6.slice/crio-eafc890e277c0aaa6f4c310a8a76926b47bbc911cc86e70d5271ca571fdab600 WatchSource:0}: Error finding container eafc890e277c0aaa6f4c310a8a76926b47bbc911cc86e70d5271ca571fdab600: Status 404 returned error can't find the container with id eafc890e277c0aaa6f4c310a8a76926b47bbc911cc86e70d5271ca571fdab600
Sep 17 17:57:17.432548 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.432464    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" event={"ID":"9116a93e-a372-46b2-8070-847defb20fe6","Type":"ContainerStarted","Data":"eafc890e277c0aaa6f4c310a8a76926b47bbc911cc86e70d5271ca571fdab600"}
Sep 17 17:57:17.434677 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.434625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" event={"ID":"8bf9b8f0-b273-4f7a-8ee4-be8df6445e18","Type":"ContainerStarted","Data":"471a36ae2b6d92138034eae2352caa06c6b786707a31d129ece063a8fd4a81ae"}
Sep 17 17:57:17.458970 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.458942    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:17.927268 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.927233    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:57:17.927478 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.927455    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Sep 17 17:57:17.927531 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:17.927512    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" podUID="8bf9b8f0-b273-4f7a-8ee4-be8df6445e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Sep 17 17:57:18.928135 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:18.928092    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Sep 17 17:57:18.928608 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:18.928156    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" podUID="8bf9b8f0-b273-4f7a-8ee4-be8df6445e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Sep 17 17:57:19.927808 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:19.927773    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Sep 17 17:57:19.927963 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:19.927833    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" podUID="8bf9b8f0-b273-4f7a-8ee4-be8df6445e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Sep 17 17:57:20.473619 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:20.473587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" event={"ID":"9116a93e-a372-46b2-8070-847defb20fe6","Type":"ContainerStarted","Data":"b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7"}
Sep 17 17:57:20.474111 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:20.473734    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:20.496981 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:20.496934    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podStartSLOduration=1.559731365 podStartE2EDuration="4.496918359s" totalImagesPullingTime="2.937186994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:20.496177039 +0000 UTC m=+1365.456062312" watchObservedRunningTime="2026-09-17 17:57:20.496918359 +0000 UTC m=+1365.456803623"
Sep 17 17:57:20.927064 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:20.927032    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Sep 17 17:57:20.927247 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:20.927085    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" podUID="8bf9b8f0-b273-4f7a-8ee4-be8df6445e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Sep 17 17:57:21.927284 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:21.927251    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Sep 17 17:57:21.927707 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:21.927302    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j" podUID="8bf9b8f0-b273-4f7a-8ee4-be8df6445e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Sep 17 17:57:22.931370 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:22.931341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:57:22.931825 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:22.931614    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:57:22.932218 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:22.932201    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j"
Sep 17 17:57:51.223932 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.223901    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"]
Sep 17 17:57:51.233198 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:57:51.233166    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:57:51.233198 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:51.233198 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:51.233198 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a" cmd=["/bin/sleep","15"]
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.233217    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" message="" err=<
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:57:51.233230    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" podUID="1b689574-0329-4705-81c2-35741e76ae82" containerName="main" containerID="cri-o://4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"
Sep 17 17:57:51.233416 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.233275    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" podUID="1b689574-0329-4705-81c2-35741e76ae82" containerName="main" containerID="cri-o://4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a" gracePeriod=60
Sep 17 17:57:51.435800 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.435762    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" podUID="1b689574-0329-4705-81c2-35741e76ae82" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Sep 17 17:57:51.480673 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.480650    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:57:51.581447 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.581420    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:51.611809 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.611782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-dshm\") pod \"1b689574-0329-4705-81c2-35741e76ae82\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") "
Sep 17 17:57:51.611809 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.611810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-model-cache\") pod \"1b689574-0329-4705-81c2-35741e76ae82\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") "
Sep 17 17:57:51.612002 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.611855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zqwm\" (UniqueName: \"kubernetes.io/projected/1b689574-0329-4705-81c2-35741e76ae82-kube-api-access-9zqwm\") pod \"1b689574-0329-4705-81c2-35741e76ae82\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") "
Sep 17 17:57:51.612002 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.611890    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-home\") pod \"1b689574-0329-4705-81c2-35741e76ae82\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") "
Sep 17 17:57:51.612002 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.611922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-tmp-dir\") pod \"1b689574-0329-4705-81c2-35741e76ae82\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") "
Sep 17 17:57:51.612158 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612135    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b689574-0329-4705-81c2-35741e76ae82" (UID: "1b689574-0329-4705-81c2-35741e76ae82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:51.612211 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612145    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b689574-0329-4705-81c2-35741e76ae82" (UID: "1b689574-0329-4705-81c2-35741e76ae82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:51.612211 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b689574-0329-4705-81c2-35741e76ae82-tls-certs\") pod \"1b689574-0329-4705-81c2-35741e76ae82\" (UID: \"1b689574-0329-4705-81c2-35741e76ae82\") "
Sep 17 17:57:51.612323 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612296    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-home" (OuterVolumeSpecName: "home") pod "1b689574-0329-4705-81c2-35741e76ae82" (UID: "1b689574-0329-4705-81c2-35741e76ae82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:51.612561 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612537    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:57:51.612785 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612564    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:57:51.612785 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.612579    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:57:51.614551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.614489    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-dshm" (OuterVolumeSpecName: "dshm") pod "1b689574-0329-4705-81c2-35741e76ae82" (UID: "1b689574-0329-4705-81c2-35741e76ae82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:51.614551 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.614514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b689574-0329-4705-81c2-35741e76ae82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b689574-0329-4705-81c2-35741e76ae82" (UID: "1b689574-0329-4705-81c2-35741e76ae82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:51.614930 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.614910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b689574-0329-4705-81c2-35741e76ae82-kube-api-access-9zqwm" (OuterVolumeSpecName: "kube-api-access-9zqwm") pod "1b689574-0329-4705-81c2-35741e76ae82" (UID: "1b689574-0329-4705-81c2-35741e76ae82"). InnerVolumeSpecName "kube-api-access-9zqwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:51.616905 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.616878    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b689574-0329-4705-81c2-35741e76ae82" containerID="4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a" exitCode=0
Sep 17 17:57:51.617012 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.616919    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"
Sep 17 17:57:51.617012 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.616950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" event={"ID":"1b689574-0329-4705-81c2-35741e76ae82","Type":"ContainerDied","Data":"4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"}
Sep 17 17:57:51.617012 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.617003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6" event={"ID":"1b689574-0329-4705-81c2-35741e76ae82","Type":"ContainerDied","Data":"4267a8a249b5f192840d9bd5086f26451a5ce7f0f2f83acd094f40bb60b4ad88"}
Sep 17 17:57:51.617168 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.617025    2561 scope.go:117] "RemoveContainer" containerID="4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"
Sep 17 17:57:51.632730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.632711    2561 scope.go:117] "RemoveContainer" containerID="4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"
Sep 17 17:57:51.632998 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:57:51.632982    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a\": container with ID starting with 4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a not found: ID does not exist" containerID="4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"
Sep 17 17:57:51.633057 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.633006    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a"} err="failed to get container status \"4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a\": rpc error: code = NotFound desc = could not find container \"4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a\": container with ID starting with 4da9c7fa07cef0930cfc1e9dd07c7aaf634b31c41a4c7b65ca9e4d7cb2f6fb9a not found: ID does not exist"
Sep 17 17:57:51.644940 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.644915    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"]
Sep 17 17:57:51.650843 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.650819    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-ncgk6"]
Sep 17 17:57:51.713592 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.713571    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b689574-0329-4705-81c2-35741e76ae82-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:57:51.713592 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.713590    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b689574-0329-4705-81c2-35741e76ae82-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:57:51.713729 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:51.713598    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zqwm\" (UniqueName: \"kubernetes.io/projected/1b689574-0329-4705-81c2-35741e76ae82-kube-api-access-9zqwm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:57:53.020646 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:53.020601    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b689574-0329-4705-81c2-35741e76ae82" path="/var/lib/kubelet/pods/1b689574-0329-4705-81c2-35741e76ae82/volumes"
Sep 17 17:57:59.021093 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:57:59.021060    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"]
Sep 17 17:58:01.479491 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:01.479456    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:11.479938 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:11.479900    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:14.042858 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:14.042783    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main" containerID="cri-o://b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7" gracePeriod=45
Sep 17 17:58:21.479672 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:21.479579    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:21.480036 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:21.479690    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:58:31.480613 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:31.480571    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:41.479915 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:41.479875    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:44.302206 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.302153    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:58:44.361747 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.361720    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz2l2\" (UniqueName: \"kubernetes.io/projected/9116a93e-a372-46b2-8070-847defb20fe6-kube-api-access-nz2l2\") pod \"9116a93e-a372-46b2-8070-847defb20fe6\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") "
Sep 17 17:58:44.361890 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.361769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9116a93e-a372-46b2-8070-847defb20fe6-tls-certs\") pod \"9116a93e-a372-46b2-8070-847defb20fe6\" (UID: \"9116a93e-a372-46b2-8070-847defb20fe6\") "
Sep 17 17:58:44.363879 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.363840    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9116a93e-a372-46b2-8070-847defb20fe6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9116a93e-a372-46b2-8070-847defb20fe6" (UID: "9116a93e-a372-46b2-8070-847defb20fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:44.363978 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.363889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9116a93e-a372-46b2-8070-847defb20fe6-kube-api-access-nz2l2" (OuterVolumeSpecName: "kube-api-access-nz2l2") pod "9116a93e-a372-46b2-8070-847defb20fe6" (UID: "9116a93e-a372-46b2-8070-847defb20fe6"). InnerVolumeSpecName "kube-api-access-nz2l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:44.462957 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.462922    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz2l2\" (UniqueName: \"kubernetes.io/projected/9116a93e-a372-46b2-8070-847defb20fe6-kube-api-access-nz2l2\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:58:44.462957 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.462947    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9116a93e-a372-46b2-8070-847defb20fe6-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:58:44.877548 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.877518    2561 generic.go:356] "Generic (PLEG): container finished" podID="9116a93e-a372-46b2-8070-847defb20fe6" containerID="b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7" exitCode=0
Sep 17 17:58:44.877730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.877559    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"
Sep 17 17:58:44.877730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.877592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" event={"ID":"9116a93e-a372-46b2-8070-847defb20fe6","Type":"ContainerDied","Data":"b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7"}
Sep 17 17:58:44.877730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.877658    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw" event={"ID":"9116a93e-a372-46b2-8070-847defb20fe6","Type":"ContainerDied","Data":"eafc890e277c0aaa6f4c310a8a76926b47bbc911cc86e70d5271ca571fdab600"}
Sep 17 17:58:44.877730 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.877688    2561 scope.go:117] "RemoveContainer" containerID="b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7"
Sep 17 17:58:44.886595 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.886574    2561 scope.go:117] "RemoveContainer" containerID="b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7"
Sep 17 17:58:44.886852 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:58:44.886834    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7\": container with ID starting with b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7 not found: ID does not exist" containerID="b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7"
Sep 17 17:58:44.886907 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.886861    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7"} err="failed to get container status \"b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7\": rpc error: code = NotFound desc = could not find container \"b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7\": container with ID starting with b0833a5ee02c6a45922fd7f344f053e840856f16dc6026a50f7294d4f23435b7 not found: ID does not exist"
Sep 17 17:58:44.898698 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.898676    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"]
Sep 17 17:58:44.904668 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:44.904642    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-9cfhw"]
Sep 17 17:58:45.020771 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:58:45.020740    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9116a93e-a372-46b2-8070-847defb20fe6" path="/var/lib/kubelet/pods/9116a93e-a372-46b2-8070-847defb20fe6/volumes"
Sep 17 17:59:02.214999 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.214962    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"]
Sep 17 17:59:02.215471 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.215437    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b689574-0329-4705-81c2-35741e76ae82" containerName="main"
Sep 17 17:59:02.215471 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.215455    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b689574-0329-4705-81c2-35741e76ae82" containerName="main"
Sep 17 17:59:02.215603 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.215492    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main"
Sep 17 17:59:02.215603 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.215501    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main"
Sep 17 17:59:02.215603 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.215598    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9116a93e-a372-46b2-8070-847defb20fe6" containerName="main"
Sep 17 17:59:02.215787 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.215610    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b689574-0329-4705-81c2-35741e76ae82" containerName="main"
Sep 17 17:59:02.218479 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.218457    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.221539 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.221519    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 17:59:02.222379 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.222361    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 17:59:02.222494 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.222478    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 17:59:02.222679 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.222666    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-nsg5j\""
Sep 17 17:59:02.230946 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.230925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"]
Sep 17 17:59:02.319532 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.319503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.319716 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.319545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.319716 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.319586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.319716 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.319603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.319716 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.319649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.319856 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.319717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljrc\" (UniqueName: \"kubernetes.io/projected/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-kube-api-access-xljrc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.420998 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.420963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.420998 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421240 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421240 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljrc\" (UniqueName: \"kubernetes.io/projected/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-kube-api-access-xljrc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421240 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421240 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421193    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421449 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421494 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.421549 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.421532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.423340 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.423316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.423582 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.423564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.432177 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.432152    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljrc\" (UniqueName: \"kubernetes.io/projected/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-kube-api-access-xljrc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fcq8b\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.529535 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.529494    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:02.803557 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.803531    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"]
Sep 17 17:59:02.805433 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:59:02.805409    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f8d52_6601_46ed_a479_3fa9a6b2eecc.slice/crio-b9d2d52844c75cf5b15b33da36b97d8e5b87bec92a0c62031ad52337fbdd63d2 WatchSource:0}: Error finding container b9d2d52844c75cf5b15b33da36b97d8e5b87bec92a0c62031ad52337fbdd63d2: Status 404 returned error can't find the container with id b9d2d52844c75cf5b15b33da36b97d8e5b87bec92a0c62031ad52337fbdd63d2
Sep 17 17:59:02.957434 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.957393    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" event={"ID":"c52f8d52-6601-46ed-a479-3fa9a6b2eecc","Type":"ContainerStarted","Data":"c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"}
Sep 17 17:59:02.957601 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.957450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" event={"ID":"c52f8d52-6601-46ed-a479-3fa9a6b2eecc","Type":"ContainerStarted","Data":"b9d2d52844c75cf5b15b33da36b97d8e5b87bec92a0c62031ad52337fbdd63d2"}
Sep 17 17:59:02.985562 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:02.985512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" podStartSLOduration=0.985495313 podStartE2EDuration="985.495313ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:02.983550334 +0000 UTC m=+1467.943435618" watchObservedRunningTime="2026-09-17 17:59:02.985495313 +0000 UTC m=+1467.945380583"
Sep 17 17:59:03.530068 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:03.530023    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:12.529951 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:12.529911    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:12.543194 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:12.543168    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:13.020486 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:13.020461    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:22.218269 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.218236    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"]
Sep 17 17:59:22.222051 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.222031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.225862 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.225839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 17:59:22.225985 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.225838    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-kg6s5\""
Sep 17 17:59:22.225985 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.225839    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-jczt2\""
Sep 17 17:59:22.225985 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.225888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 17:59:22.225985 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.225843    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 17:59:22.232462 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.232437    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"]
Sep 17 17:59:22.295000 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.294975    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f07c08a-30f0-4a40-963a-9f036c939cdd-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.295137 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.295014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n27w\" (UniqueName: \"kubernetes.io/projected/3f07c08a-30f0-4a40-963a-9f036c939cdd-kube-api-access-7n27w\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.395669 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.395622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n27w\" (UniqueName: \"kubernetes.io/projected/3f07c08a-30f0-4a40-963a-9f036c939cdd-kube-api-access-7n27w\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.395812 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.395745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f07c08a-30f0-4a40-963a-9f036c939cdd-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.398203 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.398186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f07c08a-30f0-4a40-963a-9f036c939cdd-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.406085 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.406059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n27w\" (UniqueName: \"kubernetes.io/projected/3f07c08a-30f0-4a40-963a-9f036c939cdd-kube-api-access-7n27w\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.532831 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.532806    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:22.826795 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:22.826764    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"]
Sep 17 17:59:22.828829 ip-10-0-141-122 kubenswrapper[2561]: W0917 17:59:22.828799    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f07c08a_30f0_4a40_963a_9f036c939cdd.slice/crio-4ba39d3af5a98f0945e9c2ebe7e725b81b9b8d035a72657d48e1667f83969eaa WatchSource:0}: Error finding container 4ba39d3af5a98f0945e9c2ebe7e725b81b9b8d035a72657d48e1667f83969eaa: Status 404 returned error can't find the container with id 4ba39d3af5a98f0945e9c2ebe7e725b81b9b8d035a72657d48e1667f83969eaa
Sep 17 17:59:23.055020 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:23.054982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" event={"ID":"3f07c08a-30f0-4a40-963a-9f036c939cdd","Type":"ContainerStarted","Data":"d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097"}
Sep 17 17:59:23.055206 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:23.055043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" event={"ID":"3f07c08a-30f0-4a40-963a-9f036c939cdd","Type":"ContainerStarted","Data":"4ba39d3af5a98f0945e9c2ebe7e725b81b9b8d035a72657d48e1667f83969eaa"}
Sep 17 17:59:23.055206 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:23.055084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 17:59:23.073732 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:23.073670    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podStartSLOduration=1.073650561 podStartE2EDuration="1.073650561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:23.072916886 +0000 UTC m=+1488.032802159" watchObservedRunningTime="2026-09-17 17:59:23.073650561 +0000 UTC m=+1488.033535835"
Sep 17 17:59:37.039290 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:37.039262    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:59:37.041557 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:37.041536    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 17:59:39.439084 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.439051    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"]
Sep 17 17:59:39.449184 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:59:39.449143    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:59:39.449184 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:59:39.449184 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 17:59:39.449184 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151" cmd=["/bin/sleep","15"]
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.449195    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" message="" err=<
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:59:39.449219    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" podUID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" containerName="main" containerID="cri-o://c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"
Sep 17 17:59:39.449483 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.449315    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" podUID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" containerName="main" containerID="cri-o://c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151" gracePeriod=60
Sep 17 17:59:39.790939 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.790917    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:39.844005 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.843977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tmp-dir\") pod \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") "
Sep 17 17:59:39.844179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tls-certs\") pod \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") "
Sep 17 17:59:39.844179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-model-cache\") pod \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") "
Sep 17 17:59:39.844179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844104    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xljrc\" (UniqueName: \"kubernetes.io/projected/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-kube-api-access-xljrc\") pod \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") "
Sep 17 17:59:39.844179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-home\") pod \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") "
Sep 17 17:59:39.844179 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-dshm\") pod \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\" (UID: \"c52f8d52-6601-46ed-a479-3fa9a6b2eecc\") "
Sep 17 17:59:39.844487 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c52f8d52-6601-46ed-a479-3fa9a6b2eecc" (UID: "c52f8d52-6601-46ed-a479-3fa9a6b2eecc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:39.844487 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844374    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c52f8d52-6601-46ed-a479-3fa9a6b2eecc" (UID: "c52f8d52-6601-46ed-a479-3fa9a6b2eecc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:39.844487 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844451    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-home" (OuterVolumeSpecName: "home") pod "c52f8d52-6601-46ed-a479-3fa9a6b2eecc" (UID: "c52f8d52-6601-46ed-a479-3fa9a6b2eecc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:39.844487 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.844478    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.846314 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.846291    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-dshm" (OuterVolumeSpecName: "dshm") pod "c52f8d52-6601-46ed-a479-3fa9a6b2eecc" (UID: "c52f8d52-6601-46ed-a479-3fa9a6b2eecc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:39.846314 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.846305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-kube-api-access-xljrc" (OuterVolumeSpecName: "kube-api-access-xljrc") pod "c52f8d52-6601-46ed-a479-3fa9a6b2eecc" (UID: "c52f8d52-6601-46ed-a479-3fa9a6b2eecc"). InnerVolumeSpecName "kube-api-access-xljrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:39.846658 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.846358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c52f8d52-6601-46ed-a479-3fa9a6b2eecc" (UID: "c52f8d52-6601-46ed-a479-3fa9a6b2eecc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:39.945511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.945477    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.945511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.945507    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.945511 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.945516    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljrc\" (UniqueName: \"kubernetes.io/projected/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-kube-api-access-xljrc\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.945762 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.945526    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:59:39.945762 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:39.945535    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52f8d52-6601-46ed-a479-3fa9a6b2eecc-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 17:59:40.133159 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.133127    2561 generic.go:356] "Generic (PLEG): container finished" podID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" containerID="c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151" exitCode=0
Sep 17 17:59:40.133339 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.133177    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"
Sep 17 17:59:40.133339 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.133200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" event={"ID":"c52f8d52-6601-46ed-a479-3fa9a6b2eecc","Type":"ContainerDied","Data":"c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"}
Sep 17 17:59:40.133339 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.133242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b" event={"ID":"c52f8d52-6601-46ed-a479-3fa9a6b2eecc","Type":"ContainerDied","Data":"b9d2d52844c75cf5b15b33da36b97d8e5b87bec92a0c62031ad52337fbdd63d2"}
Sep 17 17:59:40.133339 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.133264    2561 scope.go:117] "RemoveContainer" containerID="c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"
Sep 17 17:59:40.142441 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.142425    2561 scope.go:117] "RemoveContainer" containerID="c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"
Sep 17 17:59:40.142729 ip-10-0-141-122 kubenswrapper[2561]: E0917 17:59:40.142712    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151\": container with ID starting with c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151 not found: ID does not exist" containerID="c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"
Sep 17 17:59:40.142787 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.142738    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151"} err="failed to get container status \"c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151\": rpc error: code = NotFound desc = could not find container \"c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151\": container with ID starting with c3188abc84e47580fb694ca1bf0046b54512eff4eb6ac8eb441e919bf4e5f151 not found: ID does not exist"
Sep 17 17:59:40.154509 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.154485    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"]
Sep 17 17:59:40.160732 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:40.160713    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fcq8b"]
Sep 17 17:59:41.019736 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:41.019706    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" path="/var/lib/kubelet/pods/c52f8d52-6601-46ed-a479-3fa9a6b2eecc/volumes"
Sep 17 17:59:54.062067 ip-10-0-141-122 kubenswrapper[2561]: I0917 17:59:54.061989    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 18:00:00.663209 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:00.663173    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"]
Sep 17 18:00:04.061330 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:04.061292    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:14.061169 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:14.061134    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:15.682910 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:15.682858    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main" containerID="cri-o://d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097" gracePeriod=45
Sep 17 18:00:24.061861 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:24.061820    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:24.062365 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:24.061939    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 18:00:34.061513 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:34.061467    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:40.157017 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.156982    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"]
Sep 17 18:00:40.157523 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.157506    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" containerName="main"
Sep 17 18:00:40.157523 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.157525    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" containerName="main"
Sep 17 18:00:40.157677 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.157606    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52f8d52-6601-46ed-a479-3fa9a6b2eecc" containerName="main"
Sep 17 18:00:40.160336 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.160319    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.164764 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.164744    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:00:40.165698 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.165682    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:00:40.165791 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.165774    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-7smbt\""
Sep 17 18:00:40.165851 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.165783    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:00:40.180546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.180516    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"]
Sep 17 18:00:40.276815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.276786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.276962 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.276826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.276962 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.276846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz546\" (UniqueName: \"kubernetes.io/projected/8be1fb72-dd85-4385-852c-8c43196fa76a-kube-api-access-jz546\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.277034 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.276956    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.277034 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.276992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1fb72-dd85-4385-852c-8c43196fa76a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.277034 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.277011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378208 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378362 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378362 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz546\" (UniqueName: \"kubernetes.io/projected/8be1fb72-dd85-4385-852c-8c43196fa76a-kube-api-access-jz546\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378454 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378493 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1fb72-dd85-4385-852c-8c43196fa76a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378538 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378619 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378834 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378814    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.378905 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.378859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.380443 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.380423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.380837 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.380817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1fb72-dd85-4385-852c-8c43196fa76a-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.389118 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.389098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz546\" (UniqueName: \"kubernetes.io/projected/8be1fb72-dd85-4385-852c-8c43196fa76a-kube-api-access-jz546\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.470953 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.470873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:40.744487 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:40.744462    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"]
Sep 17 18:00:40.746415 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:00:40.746393    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be1fb72_dd85_4385_852c_8c43196fa76a.slice/crio-2eb97996614ef169e78154777445f6016d7a1f3d85b2e916513b2f7bdfdcf9f3 WatchSource:0}: Error finding container 2eb97996614ef169e78154777445f6016d7a1f3d85b2e916513b2f7bdfdcf9f3: Status 404 returned error can't find the container with id 2eb97996614ef169e78154777445f6016d7a1f3d85b2e916513b2f7bdfdcf9f3
Sep 17 18:00:41.414669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:41.414613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" event={"ID":"8be1fb72-dd85-4385-852c-8c43196fa76a","Type":"ContainerStarted","Data":"dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524"}
Sep 17 18:00:41.415122 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:41.414689    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" event={"ID":"8be1fb72-dd85-4385-852c-8c43196fa76a","Type":"ContainerStarted","Data":"2eb97996614ef169e78154777445f6016d7a1f3d85b2e916513b2f7bdfdcf9f3"}
Sep 17 18:00:41.433717 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:41.433658    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" podStartSLOduration=1.433612866 podStartE2EDuration="1.433612866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:41.431397869 +0000 UTC m=+1566.391283139" watchObservedRunningTime="2026-09-17 18:00:41.433612866 +0000 UTC m=+1566.393498136"
Sep 17 18:00:41.470977 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:41.470944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:44.061739 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:44.061706    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:45.927083 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:45.927055    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 18:00:46.032060 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.032033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n27w\" (UniqueName: \"kubernetes.io/projected/3f07c08a-30f0-4a40-963a-9f036c939cdd-kube-api-access-7n27w\") pod \"3f07c08a-30f0-4a40-963a-9f036c939cdd\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") "
Sep 17 18:00:46.032223 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.032105    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f07c08a-30f0-4a40-963a-9f036c939cdd-tls-certs\") pod \"3f07c08a-30f0-4a40-963a-9f036c939cdd\" (UID: \"3f07c08a-30f0-4a40-963a-9f036c939cdd\") "
Sep 17 18:00:46.034306 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.034276    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f07c08a-30f0-4a40-963a-9f036c939cdd-kube-api-access-7n27w" (OuterVolumeSpecName: "kube-api-access-7n27w") pod "3f07c08a-30f0-4a40-963a-9f036c939cdd" (UID: "3f07c08a-30f0-4a40-963a-9f036c939cdd"). InnerVolumeSpecName "kube-api-access-7n27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:46.034398 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.034381    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f07c08a-30f0-4a40-963a-9f036c939cdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f07c08a-30f0-4a40-963a-9f036c939cdd" (UID: "3f07c08a-30f0-4a40-963a-9f036c939cdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:46.133529 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.133501    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f07c08a-30f0-4a40-963a-9f036c939cdd-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:00:46.133529 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.133524    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n27w\" (UniqueName: \"kubernetes.io/projected/3f07c08a-30f0-4a40-963a-9f036c939cdd-kube-api-access-7n27w\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:00:46.438040 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.437957    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerID="d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097" exitCode=0
Sep 17 18:00:46.438040 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.438006    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"
Sep 17 18:00:46.438219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.438042    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" event={"ID":"3f07c08a-30f0-4a40-963a-9f036c939cdd","Type":"ContainerDied","Data":"d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097"}
Sep 17 18:00:46.438219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.438099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2" event={"ID":"3f07c08a-30f0-4a40-963a-9f036c939cdd","Type":"ContainerDied","Data":"4ba39d3af5a98f0945e9c2ebe7e725b81b9b8d035a72657d48e1667f83969eaa"}
Sep 17 18:00:46.438219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.438130    2561 scope.go:117] "RemoveContainer" containerID="d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097"
Sep 17 18:00:46.446719 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.446702    2561 scope.go:117] "RemoveContainer" containerID="d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097"
Sep 17 18:00:46.446976 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:00:46.446961    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097\": container with ID starting with d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097 not found: ID does not exist" containerID="d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097"
Sep 17 18:00:46.447025 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.446984    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097"} err="failed to get container status \"d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097\": rpc error: code = NotFound desc = could not find container \"d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097\": container with ID starting with d888c946fb63baff0cb6e50a993069281d4732809a889e4cadabfad4e6bc0097 not found: ID does not exist"
Sep 17 18:00:46.458967 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.458945    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"]
Sep 17 18:00:46.463696 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:46.463676    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-w7cl2"]
Sep 17 18:00:47.019717 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:47.019685    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" path="/var/lib/kubelet/pods/3f07c08a-30f0-4a40-963a-9f036c939cdd/volumes"
Sep 17 18:00:50.471929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:50.471895    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:50.484487 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:50.484465    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:51.138391 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.138356    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"]
Sep 17 18:00:51.138770 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.138750    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main"
Sep 17 18:00:51.138880 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.138810    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main"
Sep 17 18:00:51.138924 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.138912    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f07c08a-30f0-4a40-963a-9f036c939cdd" containerName="main"
Sep 17 18:00:51.145479 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.145460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.148657 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.148619    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-ckpkj\""
Sep 17 18:00:51.149801 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.149786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 18:00:51.150043 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.150027    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:00:51.150432 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.150417    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 18:00:51.175398 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175494 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175494 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175494 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175494 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175684 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175684 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncl88\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-kube-api-access-ncl88\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175684 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175684 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.175823 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.175773    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"]
Sep 17 18:00:51.275989 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.275958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276144 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.275997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276144 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncl88\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-kube-api-access-ncl88\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276144 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276144 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276144 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276401 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276497 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276561 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276561 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276561 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276764 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276849 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.276909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.276892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.278422 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.278403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.278572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.278555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.284208 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.284189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.284302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.284189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncl88\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-kube-api-access-ncl88\") pod \"router-gateway-1-openshift-default-6b46477c5b-d6v8s\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.456372 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.456298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:51.475875 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.475848    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:00:51.739714 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.739539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"]
Sep 17 18:00:51.742544 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:00:51.742507    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd08010_ea8f_4242_b5c8_960e0d23b89f.slice/crio-a4c795a86f059eb75aec1cfbee1c57dbc3741061ba5b771df0d7e0d923319791 WatchSource:0}: Error finding container a4c795a86f059eb75aec1cfbee1c57dbc3741061ba5b771df0d7e0d923319791: Status 404 returned error can't find the container with id a4c795a86f059eb75aec1cfbee1c57dbc3741061ba5b771df0d7e0d923319791
Sep 17 18:00:51.744676 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.744618    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:00:51.744765 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.744712    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:00:51.744765 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:51.744749    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:00:52.467672 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:52.467623    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s" event={"ID":"cfd08010-ea8f-4242-b5c8-960e0d23b89f","Type":"ContainerStarted","Data":"7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30"}
Sep 17 18:00:52.467863 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:52.467697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s" event={"ID":"cfd08010-ea8f-4242-b5c8-960e0d23b89f","Type":"ContainerStarted","Data":"a4c795a86f059eb75aec1cfbee1c57dbc3741061ba5b771df0d7e0d923319791"}
Sep 17 18:00:52.505607 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:52.505557    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s" podStartSLOduration=1.505542463 podStartE2EDuration="1.505542463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:52.502654967 +0000 UTC m=+1577.462540234" watchObservedRunningTime="2026-09-17 18:00:52.505542463 +0000 UTC m=+1577.465427734"
Sep 17 18:00:53.457158 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:53.457127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:53.462322 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:53.462295    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:53.472987 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:53.472967    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:00:53.473621 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:00:53.473604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:01:16.314174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.314135    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"]
Sep 17 18:01:16.319812 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.319784    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.323081 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.323051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-4pwvv\""
Sep 17 18:01:16.323383 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.323363    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:01:16.323693 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.323676    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:01:16.324098 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.324075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:01:16.332036 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.332013    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"]
Sep 17 18:01:16.397410 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397410 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdzl\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-kube-api-access-zkdzl\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397854 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397854 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.397854 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.397723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498557 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498557 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdzl\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-kube-api-access-zkdzl\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498661    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498725    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498749    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.498846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.499172 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.498987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.499226 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.499164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.499226 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.499204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.499290 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.499274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.499488 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.499470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.501164 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.501137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.501323 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.501309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.508811 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.508787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.509870 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.509847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdzl\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-kube-api-access-zkdzl\") pod \"router-gateway-1-openshift-default-7dcbd489bd-9xg6g\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.633755 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.633674    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:16.930999 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.930975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"]
Sep 17 18:01:16.933187 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:01:16.933140    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dfb897_8842_42af_b2ea_f1fbfb758fdb.slice/crio-74aae0fb483f5e18226a32314d738350d6e07729fe1edc3d58f934c2961b5438 WatchSource:0}: Error finding container 74aae0fb483f5e18226a32314d738350d6e07729fe1edc3d58f934c2961b5438: Status 404 returned error can't find the container with id 74aae0fb483f5e18226a32314d738350d6e07729fe1edc3d58f934c2961b5438
Sep 17 18:01:16.935184 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.935158    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:01:16.935254 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.935220    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:01:16.935254 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:16.935249    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:01:17.585134 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:17.585096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g" event={"ID":"29dfb897-8842-42af-b2ea-f1fbfb758fdb","Type":"ContainerStarted","Data":"09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d"}
Sep 17 18:01:17.585576 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:17.585155    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g" event={"ID":"29dfb897-8842-42af-b2ea-f1fbfb758fdb","Type":"ContainerStarted","Data":"74aae0fb483f5e18226a32314d738350d6e07729fe1edc3d58f934c2961b5438"}
Sep 17 18:01:17.608882 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:17.608836    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g" podStartSLOduration=1.6088186260000001 podStartE2EDuration="1.608818626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:17.607743216 +0000 UTC m=+1602.567628491" watchObservedRunningTime="2026-09-17 18:01:17.608818626 +0000 UTC m=+1602.568703895"
Sep 17 18:01:17.634573 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:17.634549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:18.647671 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:18.647619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:19.594307 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:19.594279    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:19.595147 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:19.595126    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:21.008810 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:21.008775    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"]
Sep 17 18:01:21.009260 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:21.009030    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s" podUID="cfd08010-ea8f-4242-b5c8-960e0d23b89f" containerName="istio-proxy" containerID="cri-o://7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30" gracePeriod=30
Sep 17 18:01:23.457769 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.457745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:01:23.569079 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569054    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-certs\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569221 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569137    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncl88\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-kube-api-access-ncl88\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569221 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-envoy\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569221 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569178    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-podinfo\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569221 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-data\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569424 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569245    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-token\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569424 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-socket\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569424 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569326    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-credential-socket\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569424 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569336    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:23.569424 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istiod-ca-cert\") pod \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\" (UID: \"cfd08010-ea8f-4242-b5c8-960e0d23b89f\") "
Sep 17 18:01:23.569822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:23.569822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569673    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:23.569822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569761    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:23.569822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569766    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-data\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.569822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:23.569822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569821    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.570143 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.569841    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-workload-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.571522 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.571497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:23.571599 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.571501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:01:23.571599 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.571545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-kube-api-access-ncl88" (OuterVolumeSpecName: "kube-api-access-ncl88") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "kube-api-access-ncl88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:23.571599 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.571558    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfd08010-ea8f-4242-b5c8-960e0d23b89f" (UID: "cfd08010-ea8f-4242-b5c8-960e0d23b89f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:23.615566 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.615538    2561 generic.go:356] "Generic (PLEG): container finished" podID="cfd08010-ea8f-4242-b5c8-960e0d23b89f" containerID="7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30" exitCode=0
Sep 17 18:01:23.615698 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.615578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s" event={"ID":"cfd08010-ea8f-4242-b5c8-960e0d23b89f","Type":"ContainerDied","Data":"7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30"}
Sep 17 18:01:23.615698 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.615584    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"
Sep 17 18:01:23.615698 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.615626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s" event={"ID":"cfd08010-ea8f-4242-b5c8-960e0d23b89f","Type":"ContainerDied","Data":"a4c795a86f059eb75aec1cfbee1c57dbc3741061ba5b771df0d7e0d923319791"}
Sep 17 18:01:23.615698 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.615677    2561 scope.go:117] "RemoveContainer" containerID="7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30"
Sep 17 18:01:23.625531 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.625511    2561 scope.go:117] "RemoveContainer" containerID="7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30"
Sep 17 18:01:23.625830 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:01:23.625813    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30\": container with ID starting with 7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30 not found: ID does not exist" containerID="7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30"
Sep 17 18:01:23.625903 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.625837    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30"} err="failed to get container status \"7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30\": rpc error: code = NotFound desc = could not find container \"7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30\": container with ID starting with 7ef725f90ecfd1928878e760908d3e3a0ce94c0ac157b4453a0e8698f26c0e30 not found: ID does not exist"
Sep 17 18:01:23.641551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.641529    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"]
Sep 17 18:01:23.645360 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.645341    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-d6v8s"]
Sep 17 18:01:23.671052 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.671031    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-token\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.671052 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.671051    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-credential-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.671163 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.671061    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istiod-ca-cert\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.671163 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.671070    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncl88\" (UniqueName: \"kubernetes.io/projected/cfd08010-ea8f-4242-b5c8-960e0d23b89f-kube-api-access-ncl88\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.671163 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.671078    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-envoy\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:23.671163 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:23.671089    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd08010-ea8f-4242-b5c8-960e0d23b89f-istio-podinfo\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:25.020744 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:25.020713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd08010-ea8f-4242-b5c8-960e0d23b89f" path="/var/lib/kubelet/pods/cfd08010-ea8f-4242-b5c8-960e0d23b89f/volumes"
Sep 17 18:01:40.511504 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:40.511464    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"]
Sep 17 18:01:40.512130 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:40.511694    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g" podUID="29dfb897-8842-42af-b2ea-f1fbfb758fdb" containerName="istio-proxy" containerID="cri-o://09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d" gracePeriod=30
Sep 17 18:01:41.927785 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:01:41.927758    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-19694-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dfb897_8842_42af_b2ea_f1fbfb758fdb.slice/crio-09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:01:42.061511 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.061453    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:42.136104 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-certs\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136225 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136126    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-socket\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136225 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136155    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-envoy\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136225 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-credential-socket\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136225 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-data\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136442 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136258    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istiod-ca-cert\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136442 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-token\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136442 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136329    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-podinfo\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136442 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136370    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkdzl\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-kube-api-access-zkdzl\") pod \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\" (UID: \"29dfb897-8842-42af-b2ea-f1fbfb758fdb\") "
Sep 17 18:01:42.136442 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136375    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:42.136845 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:42.136845 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:42.136964 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.136927    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:42.137294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.137047    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istiod-ca-cert\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.137294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.137068    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.137294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.137081    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-workload-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.137294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.137093    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-data\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.137294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.137255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:42.138805 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.138777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:42.138902 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.138783    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:01:42.138954 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.138915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:42.139197 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.139174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-kube-api-access-zkdzl" (OuterVolumeSpecName: "kube-api-access-zkdzl") pod "29dfb897-8842-42af-b2ea-f1fbfb758fdb" (UID: "29dfb897-8842-42af-b2ea-f1fbfb758fdb"). InnerVolumeSpecName "kube-api-access-zkdzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:42.238351 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.238319    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-podinfo\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.238351 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.238349    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdzl\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-kube-api-access-zkdzl\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.238551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.238363    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-envoy\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.238551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.238374    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29dfb897-8842-42af-b2ea-f1fbfb758fdb-credential-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.238551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.238388    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29dfb897-8842-42af-b2ea-f1fbfb758fdb-istio-token\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:01:42.710057 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.709969    2561 generic.go:356] "Generic (PLEG): container finished" podID="29dfb897-8842-42af-b2ea-f1fbfb758fdb" containerID="09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d" exitCode=0
Sep 17 18:01:42.710057 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.710018    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"
Sep 17 18:01:42.710232 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.710051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g" event={"ID":"29dfb897-8842-42af-b2ea-f1fbfb758fdb","Type":"ContainerDied","Data":"09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d"}
Sep 17 18:01:42.710232 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.710112    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g" event={"ID":"29dfb897-8842-42af-b2ea-f1fbfb758fdb","Type":"ContainerDied","Data":"74aae0fb483f5e18226a32314d738350d6e07729fe1edc3d58f934c2961b5438"}
Sep 17 18:01:42.710232 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.710142    2561 scope.go:117] "RemoveContainer" containerID="09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d"
Sep 17 18:01:42.725174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.725155    2561 scope.go:117] "RemoveContainer" containerID="09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d"
Sep 17 18:01:42.725524 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:01:42.725498    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d\": container with ID starting with 09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d not found: ID does not exist" containerID="09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d"
Sep 17 18:01:42.725621 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.725530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d"} err="failed to get container status \"09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d\": rpc error: code = NotFound desc = could not find container \"09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d\": container with ID starting with 09e35a3e6b11ce4d53500c0b70ad44146f5043e60d6a83e073a47967bf42736d not found: ID does not exist"
Sep 17 18:01:42.735559 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.735534    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"]
Sep 17 18:01:42.739658 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:42.739622    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-9xg6g"]
Sep 17 18:01:43.020454 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:43.020421    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29dfb897-8842-42af-b2ea-f1fbfb758fdb" path="/var/lib/kubelet/pods/29dfb897-8842-42af-b2ea-f1fbfb758fdb/volumes"
Sep 17 18:01:46.097083 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.096998    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"]
Sep 17 18:01:46.097589 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.097478    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dfb897-8842-42af-b2ea-f1fbfb758fdb" containerName="istio-proxy"
Sep 17 18:01:46.097589 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.097497    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dfb897-8842-42af-b2ea-f1fbfb758fdb" containerName="istio-proxy"
Sep 17 18:01:46.097589 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.097525    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd08010-ea8f-4242-b5c8-960e0d23b89f" containerName="istio-proxy"
Sep 17 18:01:46.097589 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.097533    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd08010-ea8f-4242-b5c8-960e0d23b89f" containerName="istio-proxy"
Sep 17 18:01:46.097833 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.097622    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd08010-ea8f-4242-b5c8-960e0d23b89f" containerName="istio-proxy"
Sep 17 18:01:46.097833 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.097668    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dfb897-8842-42af-b2ea-f1fbfb758fdb" containerName="istio-proxy"
Sep 17 18:01:46.101707 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.101684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.105439 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.105410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-2p96l\""
Sep 17 18:01:46.105566 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.105500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:01:46.105702 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.105679    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:01:46.105893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.105875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-txq77\""
Sep 17 18:01:46.106068 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.106052    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:01:46.112723 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.112703    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"]
Sep 17 18:01:46.172501 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.172470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.172616 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.172557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjp48\" (UniqueName: \"kubernetes.io/projected/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-kube-api-access-jjp48\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.273941 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.273912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjp48\" (UniqueName: \"kubernetes.io/projected/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-kube-api-access-jjp48\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.274088 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.273963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.276369 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.276349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.290562 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.290531    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjp48\" (UniqueName: \"kubernetes.io/projected/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-kube-api-access-jjp48\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.413820 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.413733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:46.689345 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.689320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"]
Sep 17 18:01:46.691535 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:01:46.691511    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3bd23b_15dc_4aad_8770_1c7c0f70a9ab.slice/crio-8df89b04b3257d626e2f86eeb27d7354c71de0f7d5ba96ea744640ae0a312143 WatchSource:0}: Error finding container 8df89b04b3257d626e2f86eeb27d7354c71de0f7d5ba96ea744640ae0a312143: Status 404 returned error can't find the container with id 8df89b04b3257d626e2f86eeb27d7354c71de0f7d5ba96ea744640ae0a312143
Sep 17 18:01:46.730842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:46.730816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" event={"ID":"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab","Type":"ContainerStarted","Data":"8df89b04b3257d626e2f86eeb27d7354c71de0f7d5ba96ea744640ae0a312143"}
Sep 17 18:01:47.739802 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:47.739767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" event={"ID":"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab","Type":"ContainerStarted","Data":"1165fa6af334e1b12dd8490a00bc2457a59b6f3222096fc5ea48b7b7929570e5"}
Sep 17 18:01:47.740196 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:47.739844    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:01:47.770202 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:01:47.770143    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" podStartSLOduration=1.770126808 podStartE2EDuration="1.770126808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:47.769549502 +0000 UTC m=+1632.729434779" watchObservedRunningTime="2026-09-17 18:01:47.770126808 +0000 UTC m=+1632.730012077"
Sep 17 18:02:18.746455 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:18.746426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:02:26.664958 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:26.664920    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"]
Sep 17 18:02:26.674225 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:02:26.674191    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:02:26.674225 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:26.674225 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:26.674225 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524" cmd=["/bin/sleep","15"]
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:26.674241    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" message="" err=<
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:02:26.674253    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" podUID="8be1fb72-dd85-4385-852c-8c43196fa76a" containerName="main" containerID="cri-o://dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524"
Sep 17 18:02:26.674471 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:26.674286    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" podUID="8be1fb72-dd85-4385-852c-8c43196fa76a" containerName="main" containerID="cri-o://dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524" gracePeriod=60
Sep 17 18:02:26.921035 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:26.920962    2561 generic.go:356] "Generic (PLEG): container finished" podID="8be1fb72-dd85-4385-852c-8c43196fa76a" containerID="dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524" exitCode=0
Sep 17 18:02:26.921165 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:26.921040    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" event={"ID":"8be1fb72-dd85-4385-852c-8c43196fa76a","Type":"ContainerDied","Data":"dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524"}
Sep 17 18:02:27.027842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.027818    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:02:27.130697 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130666    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-home\") pod \"8be1fb72-dd85-4385-852c-8c43196fa76a\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") "
Sep 17 18:02:27.130893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130747    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz546\" (UniqueName: \"kubernetes.io/projected/8be1fb72-dd85-4385-852c-8c43196fa76a-kube-api-access-jz546\") pod \"8be1fb72-dd85-4385-852c-8c43196fa76a\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") "
Sep 17 18:02:27.130893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-model-cache\") pod \"8be1fb72-dd85-4385-852c-8c43196fa76a\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") "
Sep 17 18:02:27.130893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-dshm\") pod \"8be1fb72-dd85-4385-852c-8c43196fa76a\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") "
Sep 17 18:02:27.130893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-tmp-dir\") pod \"8be1fb72-dd85-4385-852c-8c43196fa76a\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") "
Sep 17 18:02:27.130893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130845    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1fb72-dd85-4385-852c-8c43196fa76a-tls-certs\") pod \"8be1fb72-dd85-4385-852c-8c43196fa76a\" (UID: \"8be1fb72-dd85-4385-852c-8c43196fa76a\") "
Sep 17 18:02:27.131168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.130961    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-home" (OuterVolumeSpecName: "home") pod "8be1fb72-dd85-4385-852c-8c43196fa76a" (UID: "8be1fb72-dd85-4385-852c-8c43196fa76a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.131168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.131083    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8be1fb72-dd85-4385-852c-8c43196fa76a" (UID: "8be1fb72-dd85-4385-852c-8c43196fa76a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.131168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.131121    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.131168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.131144    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8be1fb72-dd85-4385-852c-8c43196fa76a" (UID: "8be1fb72-dd85-4385-852c-8c43196fa76a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.133062 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.133034    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-dshm" (OuterVolumeSpecName: "dshm") pod "8be1fb72-dd85-4385-852c-8c43196fa76a" (UID: "8be1fb72-dd85-4385-852c-8c43196fa76a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.133183 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.133085    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be1fb72-dd85-4385-852c-8c43196fa76a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8be1fb72-dd85-4385-852c-8c43196fa76a" (UID: "8be1fb72-dd85-4385-852c-8c43196fa76a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:27.133183 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.133137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be1fb72-dd85-4385-852c-8c43196fa76a-kube-api-access-jz546" (OuterVolumeSpecName: "kube-api-access-jz546") pod "8be1fb72-dd85-4385-852c-8c43196fa76a" (UID: "8be1fb72-dd85-4385-852c-8c43196fa76a"). InnerVolumeSpecName "kube-api-access-jz546". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:27.231722 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.231692    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz546\" (UniqueName: \"kubernetes.io/projected/8be1fb72-dd85-4385-852c-8c43196fa76a-kube-api-access-jz546\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.231722 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.231722    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.231913 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.231734    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.231913 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.231746    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8be1fb72-dd85-4385-852c-8c43196fa76a-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.231913 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.231756    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8be1fb72-dd85-4385-852c-8c43196fa76a-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.932225 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.932197    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"
Sep 17 18:02:27.932673 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.932191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h" event={"ID":"8be1fb72-dd85-4385-852c-8c43196fa76a","Type":"ContainerDied","Data":"2eb97996614ef169e78154777445f6016d7a1f3d85b2e916513b2f7bdfdcf9f3"}
Sep 17 18:02:27.932673 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.932357    2561 scope.go:117] "RemoveContainer" containerID="dc88dbf2111f4d889b3fc8aa7fa5e9129659b29ec09ac1201fcffde4b281a524"
Sep 17 18:02:27.953404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.953377    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"]
Sep 17 18:02:27.958463 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:27.958442    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-b289h"]
Sep 17 18:02:29.021867 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:02:29.021837    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be1fb72-dd85-4385-852c-8c43196fa76a" path="/var/lib/kubelet/pods/8be1fb72-dd85-4385-852c-8c43196fa76a/volumes"
Sep 17 18:03:31.355094 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.355056    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"]
Sep 17 18:03:31.355579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.355551    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be1fb72-dd85-4385-852c-8c43196fa76a" containerName="main"
Sep 17 18:03:31.355579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.355566    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be1fb72-dd85-4385-852c-8c43196fa76a" containerName="main"
Sep 17 18:03:31.355692 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.355683    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be1fb72-dd85-4385-852c-8c43196fa76a" containerName="main"
Sep 17 18:03:31.358896 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.358872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.361773 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.361752    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:03:31.363076 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.363055    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:03:31.363146 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.363060    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-zcqd8\""
Sep 17 18:03:31.363146 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.363117    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:03:31.392251 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.392228    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"]
Sep 17 18:03:31.475584 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.475536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.475584 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.475595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8b2\" (UniqueName: \"kubernetes.io/projected/199306ad-f6fc-4a95-90a2-70bc62f15ec9-kube-api-access-pq8b2\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.475929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.475707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.475929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.475784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.475929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.475846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.475929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.475876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.576809 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.576773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8b2\" (UniqueName: \"kubernetes.io/projected/199306ad-f6fc-4a95-90a2-70bc62f15ec9-kube-api-access-pq8b2\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.576991 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.576815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.576991 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.576862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.576991 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.576917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.576991 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.576944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.577216 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.576989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.577306 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.577288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.577370 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.577347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.577417 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.577401    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.579369 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.579348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.579535 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.579516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.585669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.585626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8b2\" (UniqueName: \"kubernetes.io/projected/199306ad-f6fc-4a95-90a2-70bc62f15ec9-kube-api-access-pq8b2\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.669547 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.669470    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:31.944448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:31.944116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"]
Sep 17 18:03:31.946812 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:03:31.946782    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199306ad_f6fc_4a95_90a2_70bc62f15ec9.slice/crio-cf9998d7890214dab5e393846700f11dbb07a366ad10d0075f11c35ae0c0355d WatchSource:0}: Error finding container cf9998d7890214dab5e393846700f11dbb07a366ad10d0075f11c35ae0c0355d: Status 404 returned error can't find the container with id cf9998d7890214dab5e393846700f11dbb07a366ad10d0075f11c35ae0c0355d
Sep 17 18:03:32.223404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:32.223312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" event={"ID":"199306ad-f6fc-4a95-90a2-70bc62f15ec9","Type":"ContainerStarted","Data":"b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"}
Sep 17 18:03:32.223404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:32.223368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" event={"ID":"199306ad-f6fc-4a95-90a2-70bc62f15ec9","Type":"ContainerStarted","Data":"cf9998d7890214dab5e393846700f11dbb07a366ad10d0075f11c35ae0c0355d"}
Sep 17 18:03:32.243367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:32.243314    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" podStartSLOduration=1.243299314 podStartE2EDuration="1.243299314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:32.242140171 +0000 UTC m=+1737.202025442" watchObservedRunningTime="2026-09-17 18:03:32.243299314 +0000 UTC m=+1737.203184583"
Sep 17 18:03:32.670546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:32.670514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:41.669924 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:41.669829    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:41.682360 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:41.682331    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:03:42.280581 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:03:42.280552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:04:07.188100 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.188065    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"]
Sep 17 18:04:07.196987 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:04:07.196955    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:04:07.196987 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:07.196987 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:07.196987 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043" cmd=["/bin/sleep","15"]
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.197002    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" message="" err=<
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:04:07.197014    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerName="main" containerID="cri-o://b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"
Sep 17 18:04:07.197322 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.197059    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerName="main" containerID="cri-o://b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043" gracePeriod=60
Sep 17 18:04:07.269821 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.269783    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 17 18:04:07.544932 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.544908    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:04:07.590101 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590075    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq8b2\" (UniqueName: \"kubernetes.io/projected/199306ad-f6fc-4a95-90a2-70bc62f15ec9-kube-api-access-pq8b2\") pod \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") "
Sep 17 18:04:07.590266 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590111    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tls-certs\") pod \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") "
Sep 17 18:04:07.590266 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-home\") pod \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") "
Sep 17 18:04:07.590266 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-model-cache\") pod \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") "
Sep 17 18:04:07.590425 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590264    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-dshm\") pod \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") "
Sep 17 18:04:07.590425 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590292    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tmp-dir\") pod \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\" (UID: \"199306ad-f6fc-4a95-90a2-70bc62f15ec9\") "
Sep 17 18:04:07.590534 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590427    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-home" (OuterVolumeSpecName: "home") pod "199306ad-f6fc-4a95-90a2-70bc62f15ec9" (UID: "199306ad-f6fc-4a95-90a2-70bc62f15ec9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.590534 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-model-cache" (OuterVolumeSpecName: "model-cache") pod "199306ad-f6fc-4a95-90a2-70bc62f15ec9" (UID: "199306ad-f6fc-4a95-90a2-70bc62f15ec9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.590670 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590643    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.590670 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590661    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.590770 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.590714    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "199306ad-f6fc-4a95-90a2-70bc62f15ec9" (UID: "199306ad-f6fc-4a95-90a2-70bc62f15ec9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.592302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.592283    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "199306ad-f6fc-4a95-90a2-70bc62f15ec9" (UID: "199306ad-f6fc-4a95-90a2-70bc62f15ec9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:07.592694 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.592670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-dshm" (OuterVolumeSpecName: "dshm") pod "199306ad-f6fc-4a95-90a2-70bc62f15ec9" (UID: "199306ad-f6fc-4a95-90a2-70bc62f15ec9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.592786 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.592694    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199306ad-f6fc-4a95-90a2-70bc62f15ec9-kube-api-access-pq8b2" (OuterVolumeSpecName: "kube-api-access-pq8b2") pod "199306ad-f6fc-4a95-90a2-70bc62f15ec9" (UID: "199306ad-f6fc-4a95-90a2-70bc62f15ec9"). InnerVolumeSpecName "kube-api-access-pq8b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:07.692051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.692010    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.692051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.692050    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.692261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.692065    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq8b2\" (UniqueName: \"kubernetes.io/projected/199306ad-f6fc-4a95-90a2-70bc62f15ec9-kube-api-access-pq8b2\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.692261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:07.692081    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199306ad-f6fc-4a95-90a2-70bc62f15ec9-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:04:08.385289 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.385259    2561 generic.go:356] "Generic (PLEG): container finished" podID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerID="b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043" exitCode=0
Sep 17 18:04:08.385780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.385304    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"
Sep 17 18:04:08.385780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.385349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" event={"ID":"199306ad-f6fc-4a95-90a2-70bc62f15ec9","Type":"ContainerDied","Data":"b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"}
Sep 17 18:04:08.385780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.385417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np" event={"ID":"199306ad-f6fc-4a95-90a2-70bc62f15ec9","Type":"ContainerDied","Data":"cf9998d7890214dab5e393846700f11dbb07a366ad10d0075f11c35ae0c0355d"}
Sep 17 18:04:08.385780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.385452    2561 scope.go:117] "RemoveContainer" containerID="b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"
Sep 17 18:04:08.400366 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.400349    2561 scope.go:117] "RemoveContainer" containerID="b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"
Sep 17 18:04:08.400619 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:04:08.400601    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043\": container with ID starting with b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043 not found: ID does not exist" containerID="b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"
Sep 17 18:04:08.400705 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.400650    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043"} err="failed to get container status \"b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043\": rpc error: code = NotFound desc = could not find container \"b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043\": container with ID starting with b064d87b7d4556f2c922934465d80664a79db4aa013f3b5abd11dfb6f50c9043 not found: ID does not exist"
Sep 17 18:04:08.413005 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.412981    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"]
Sep 17 18:04:08.416642 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:08.416615    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444d28np"]
Sep 17 18:04:09.019832 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:09.019802    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" path="/var/lib/kubelet/pods/199306ad-f6fc-4a95-90a2-70bc62f15ec9/volumes"
Sep 17 18:04:31.480458 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:31.480412    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"]
Sep 17 18:04:37.066470 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:37.066444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:04:37.069607 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:37.069588    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:04:38.746952 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:38.746919    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:46.500298 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:46.500244    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main" containerID="cri-o://1165fa6af334e1b12dd8490a00bc2457a59b6f3222096fc5ea48b7b7929570e5" gracePeriod=45
Sep 17 18:04:48.746211 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:48.746177    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:58.746058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:58.746022    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:58.746439 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:04:58.746130    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:05:08.746369 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:08.746333    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:05:16.693807 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.693777    2561 generic.go:356] "Generic (PLEG): container finished" podID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerID="1165fa6af334e1b12dd8490a00bc2457a59b6f3222096fc5ea48b7b7929570e5" exitCode=0
Sep 17 18:05:16.694155 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.693846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" event={"ID":"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab","Type":"ContainerDied","Data":"1165fa6af334e1b12dd8490a00bc2457a59b6f3222096fc5ea48b7b7929570e5"}
Sep 17 18:05:16.742905 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.742881    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:05:16.878168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.878073    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-tls-certs\") pod \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") "
Sep 17 18:05:16.878168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.878116    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjp48\" (UniqueName: \"kubernetes.io/projected/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-kube-api-access-jjp48\") pod \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\" (UID: \"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab\") "
Sep 17 18:05:16.880300 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.880266    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" (UID: "be3bd23b-15dc-4aad-8770-1c7c0f70a9ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:16.880399 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.880325    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-kube-api-access-jjp48" (OuterVolumeSpecName: "kube-api-access-jjp48") pod "be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" (UID: "be3bd23b-15dc-4aad-8770-1c7c0f70a9ab"). InnerVolumeSpecName "kube-api-access-jjp48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:16.979450 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.979417    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:05:16.979450 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:16.979444    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjp48\" (UniqueName: \"kubernetes.io/projected/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab-kube-api-access-jjp48\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:05:17.699693 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:17.699658    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"
Sep 17 18:05:17.700158 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:17.699664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm" event={"ID":"be3bd23b-15dc-4aad-8770-1c7c0f70a9ab","Type":"ContainerDied","Data":"8df89b04b3257d626e2f86eeb27d7354c71de0f7d5ba96ea744640ae0a312143"}
Sep 17 18:05:17.700158 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:17.699806    2561 scope.go:117] "RemoveContainer" containerID="1165fa6af334e1b12dd8490a00bc2457a59b6f3222096fc5ea48b7b7929570e5"
Sep 17 18:05:17.717961 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:17.717936    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"]
Sep 17 18:05:17.722117 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:17.722095    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-8gpbm"]
Sep 17 18:05:19.020126 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:05:19.020096    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" path="/var/lib/kubelet/pods/be3bd23b-15dc-4aad-8770-1c7c0f70a9ab/volumes"
Sep 17 18:06:12.407154 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407122    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"]
Sep 17 18:06:12.407771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407459    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main"
Sep 17 18:06:12.407771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407473    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main"
Sep 17 18:06:12.407771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407496    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerName="main"
Sep 17 18:06:12.407771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407501    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerName="main"
Sep 17 18:06:12.407771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407555    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3bd23b-15dc-4aad-8770-1c7c0f70a9ab" containerName="main"
Sep 17 18:06:12.407771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.407564    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="199306ad-f6fc-4a95-90a2-70bc62f15ec9" containerName="main"
Sep 17 18:06:12.410760 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.410737    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.414663 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.414622    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 18:06:12.414787 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.414690    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 18:06:12.414787 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.414704    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-jf7jg\""
Sep 17 18:06:12.414787 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.414720    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 18:06:12.414787 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.414691    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-k5b8m\""
Sep 17 18:06:12.423130 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.423107    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"]
Sep 17 18:06:12.547779 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.547744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhr6x\" (UniqueName: \"kubernetes.io/projected/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-kube-api-access-bhr6x\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.547984 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.547869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.649130 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.649102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.649282 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.649143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhr6x\" (UniqueName: \"kubernetes.io/projected/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-kube-api-access-bhr6x\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.651600 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.651581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.676228 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.676162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhr6x\" (UniqueName: \"kubernetes.io/projected/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-kube-api-access-bhr6x\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.720771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.720744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:12.993657 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:12.993540    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"]
Sep 17 18:06:12.996436 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:06:12.996400    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56d4f88_c4d1_40f1_bde6_e0e96a54d69a.slice/crio-1ffa58066d5388e958e3da640e9c09904a88a7ba2676c280ad783c6ee265ae57 WatchSource:0}: Error finding container 1ffa58066d5388e958e3da640e9c09904a88a7ba2676c280ad783c6ee265ae57: Status 404 returned error can't find the container with id 1ffa58066d5388e958e3da640e9c09904a88a7ba2676c280ad783c6ee265ae57
Sep 17 18:06:13.951285 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:13.951249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" event={"ID":"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a","Type":"ContainerStarted","Data":"e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061"}
Sep 17 18:06:13.951695 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:13.951333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" event={"ID":"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a","Type":"ContainerStarted","Data":"1ffa58066d5388e958e3da640e9c09904a88a7ba2676c280ad783c6ee265ae57"}
Sep 17 18:06:13.951695 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:13.951357    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:06:13.970367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:13.970318    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podStartSLOduration=1.970304955 podStartE2EDuration="1.970304955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:13.96941594 +0000 UTC m=+1898.929301209" watchObservedRunningTime="2026-09-17 18:06:13.970304955 +0000 UTC m=+1898.930190224"
Sep 17 18:06:44.957805 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:06:44.957772    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:07:26.867423 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.867340    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"]
Sep 17 18:07:26.870855 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.870833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:26.875777 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.875753    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:07:26.875900 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.875753    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:07:26.875900 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.875754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-fsg6t\""
Sep 17 18:07:26.875900 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.875891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-z2wc7\""
Sep 17 18:07:26.879047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.879022    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:07:26.896276 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.896251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"]
Sep 17 18:07:26.975922 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.975891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:26.976068 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:26.975947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr67q\" (UniqueName: \"kubernetes.io/projected/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-kube-api-access-gr67q\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:27.077118 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:27.077081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:27.077118 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:27.077115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr67q\" (UniqueName: \"kubernetes.io/projected/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-kube-api-access-gr67q\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:27.079576 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:27.079551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:27.088559 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:27.088536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr67q\" (UniqueName: \"kubernetes.io/projected/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-kube-api-access-gr67q\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:27.181414 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:27.181333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:27.452354 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:27.452273    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"]
Sep 17 18:07:27.455532 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:07:27.455501    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cc1c90_58f8_4133_ad1f_8cdf9fad2530.slice/crio-805f41f73e462fbda91cd2590da24a615987c5134e85cd933d0b6b68f847fcbe WatchSource:0}: Error finding container 805f41f73e462fbda91cd2590da24a615987c5134e85cd933d0b6b68f847fcbe: Status 404 returned error can't find the container with id 805f41f73e462fbda91cd2590da24a615987c5134e85cd933d0b6b68f847fcbe
Sep 17 18:07:28.286517 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:28.286486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" event={"ID":"79cc1c90-58f8-4133-ad1f-8cdf9fad2530","Type":"ContainerStarted","Data":"16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194"}
Sep 17 18:07:28.286926 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:28.286536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" event={"ID":"79cc1c90-58f8-4133-ad1f-8cdf9fad2530","Type":"ContainerStarted","Data":"805f41f73e462fbda91cd2590da24a615987c5134e85cd933d0b6b68f847fcbe"}
Sep 17 18:07:28.286926 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:28.286598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:07:28.312463 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:28.312418    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podStartSLOduration=2.312403169 podStartE2EDuration="2.312403169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:28.311072554 +0000 UTC m=+1973.270957826" watchObservedRunningTime="2026-09-17 18:07:28.312403169 +0000 UTC m=+1973.272288443"
Sep 17 18:07:59.293440 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:07:59.293414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:08:04.363067 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:04.363032    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"]
Sep 17 18:08:09.293054 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:09.293019    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:19.293325 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:19.293283    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:19.383757 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:19.383719    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main" containerID="cri-o://16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194" gracePeriod=45
Sep 17 18:08:29.293887 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:29.293850    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:29.294351 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:29.293971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:08:39.293190 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:39.293151    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:44.061738 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:44.061701    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"]
Sep 17 18:08:44.958651 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:44.958593    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:49.293281 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.293194    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:49.625066 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.625040    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:08:49.660389 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.660354    2561 generic.go:356] "Generic (PLEG): container finished" podID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerID="16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194" exitCode=0
Sep 17 18:08:49.660533 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.660395    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"
Sep 17 18:08:49.660533 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.660435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" event={"ID":"79cc1c90-58f8-4133-ad1f-8cdf9fad2530","Type":"ContainerDied","Data":"16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194"}
Sep 17 18:08:49.660533 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.660521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps" event={"ID":"79cc1c90-58f8-4133-ad1f-8cdf9fad2530","Type":"ContainerDied","Data":"805f41f73e462fbda91cd2590da24a615987c5134e85cd933d0b6b68f847fcbe"}
Sep 17 18:08:49.660722 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.660551    2561 scope.go:117] "RemoveContainer" containerID="16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194"
Sep 17 18:08:49.668860 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.668840    2561 scope.go:117] "RemoveContainer" containerID="16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194"
Sep 17 18:08:49.669123 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:08:49.669104    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194\": container with ID starting with 16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194 not found: ID does not exist" containerID="16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194"
Sep 17 18:08:49.669168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.669129    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194"} err="failed to get container status \"16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194\": rpc error: code = NotFound desc = could not find container \"16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194\": container with ID starting with 16d0dcc8ebc3c1cd16b903b0b575e78ed2db6488d2f4cdcf92254f1eb6e58194 not found: ID does not exist"
Sep 17 18:08:49.769398 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.769376    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-tls-certs\") pod \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") "
Sep 17 18:08:49.769531 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.769424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr67q\" (UniqueName: \"kubernetes.io/projected/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-kube-api-access-gr67q\") pod \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\" (UID: \"79cc1c90-58f8-4133-ad1f-8cdf9fad2530\") "
Sep 17 18:08:49.771525 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.771497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79cc1c90-58f8-4133-ad1f-8cdf9fad2530" (UID: "79cc1c90-58f8-4133-ad1f-8cdf9fad2530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:49.771662 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.771617    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-kube-api-access-gr67q" (OuterVolumeSpecName: "kube-api-access-gr67q") pod "79cc1c90-58f8-4133-ad1f-8cdf9fad2530" (UID: "79cc1c90-58f8-4133-ad1f-8cdf9fad2530"). InnerVolumeSpecName "kube-api-access-gr67q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:49.871009 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.870929    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:08:49.871009 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.870957    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr67q\" (UniqueName: \"kubernetes.io/projected/79cc1c90-58f8-4133-ad1f-8cdf9fad2530-kube-api-access-gr67q\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:08:49.982745 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.982716    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"]
Sep 17 18:08:49.986734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:49.986707    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785h64ps"]
Sep 17 18:08:51.021730 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:51.021692    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" path="/var/lib/kubelet/pods/79cc1c90-58f8-4133-ad1f-8cdf9fad2530/volumes"
Sep 17 18:08:54.958478 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:54.958435    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:59.080732 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:08:59.080689    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main" containerID="cri-o://e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061" gracePeriod=45
Sep 17 18:09:04.957886 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:04.957849    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:09:04.958303 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:04.957944    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:09:14.958158 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:14.958125    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:09:24.957708 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:24.957669    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:09:29.338107 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.338048    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:09:29.396072 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.396040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-tls-certs\") pod \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") "
Sep 17 18:09:29.396072 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.396074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhr6x\" (UniqueName: \"kubernetes.io/projected/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-kube-api-access-bhr6x\") pod \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\" (UID: \"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a\") "
Sep 17 18:09:29.398275 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.398234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" (UID: "c56d4f88-c4d1-40f1-bde6-e0e96a54d69a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:29.398394 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.398336    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-kube-api-access-bhr6x" (OuterVolumeSpecName: "kube-api-access-bhr6x") pod "c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" (UID: "c56d4f88-c4d1-40f1-bde6-e0e96a54d69a"). InnerVolumeSpecName "kube-api-access-bhr6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:29.497593 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.497561    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:09:29.497593 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.497588    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhr6x\" (UniqueName: \"kubernetes.io/projected/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a-kube-api-access-bhr6x\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:09:29.844772 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.844741    2561 generic.go:356] "Generic (PLEG): container finished" podID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerID="e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061" exitCode=0
Sep 17 18:09:29.844930 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.844799    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"
Sep 17 18:09:29.844930 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.844802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" event={"ID":"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a","Type":"ContainerDied","Data":"e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061"}
Sep 17 18:09:29.845047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.844936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5" event={"ID":"c56d4f88-c4d1-40f1-bde6-e0e96a54d69a","Type":"ContainerDied","Data":"1ffa58066d5388e958e3da640e9c09904a88a7ba2676c280ad783c6ee265ae57"}
Sep 17 18:09:29.845047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.844959    2561 scope.go:117] "RemoveContainer" containerID="e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061"
Sep 17 18:09:29.853660 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.853625    2561 scope.go:117] "RemoveContainer" containerID="e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061"
Sep 17 18:09:29.853940 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:09:29.853923    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061\": container with ID starting with e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061 not found: ID does not exist" containerID="e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061"
Sep 17 18:09:29.853988 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.853947    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061"} err="failed to get container status \"e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061\": rpc error: code = NotFound desc = could not find container \"e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061\": container with ID starting with e2df15af7ac09758627021d57793099fd4dcb7b1d545af2e6712cd9926e2e061 not found: ID does not exist"
Sep 17 18:09:29.866065 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.866041    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"]
Sep 17 18:09:29.869875 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:29.869857    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cclpr5"]
Sep 17 18:09:31.019238 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:31.019205    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" path="/var/lib/kubelet/pods/c56d4f88-c4d1-40f1-bde6-e0e96a54d69a/volumes"
Sep 17 18:09:37.093707 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:37.093680    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:09:37.096756 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:37.096735    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:09:45.267874 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.267840    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"]
Sep 17 18:09:45.268461 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.268312    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main"
Sep 17 18:09:45.268461 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.268332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main"
Sep 17 18:09:45.268461 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.268360    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main"
Sep 17 18:09:45.268461 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.268369    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main"
Sep 17 18:09:45.268749 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.268482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56d4f88-c4d1-40f1-bde6-e0e96a54d69a" containerName="main"
Sep 17 18:09:45.268749 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.268498    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79cc1c90-58f8-4133-ad1f-8cdf9fad2530" containerName="main"
Sep 17 18:09:45.272868 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.272846    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.276528 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.276499    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:09:45.276711 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.276550    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:09:45.276711 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.276648    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-22zv4\""
Sep 17 18:09:45.276842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.276709    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-45zt2\""
Sep 17 18:09:45.276842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.276773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:09:45.282732 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.282710    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"]
Sep 17 18:09:45.327757 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.327735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57kp\" (UniqueName: \"kubernetes.io/projected/b19561a3-053d-4ad5-9503-4650530db8a4-kube-api-access-p57kp\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.327873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.327779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19561a3-053d-4ad5-9503-4650530db8a4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.428471 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.428439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57kp\" (UniqueName: \"kubernetes.io/projected/b19561a3-053d-4ad5-9503-4650530db8a4-kube-api-access-p57kp\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.428605 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.428489    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19561a3-053d-4ad5-9503-4650530db8a4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.430946 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.430922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19561a3-053d-4ad5-9503-4650530db8a4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.438477 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.438453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57kp\" (UniqueName: \"kubernetes.io/projected/b19561a3-053d-4ad5-9503-4650530db8a4-kube-api-access-p57kp\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.583591 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.583498    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:45.856988 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.856913    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"]
Sep 17 18:09:45.860573 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:09:45.860545    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19561a3_053d_4ad5_9503_4650530db8a4.slice/crio-3919aed3296e4e816d6989d53c57da15ba904837019ac8bf9c4f1f63656f8d26 WatchSource:0}: Error finding container 3919aed3296e4e816d6989d53c57da15ba904837019ac8bf9c4f1f63656f8d26: Status 404 returned error can't find the container with id 3919aed3296e4e816d6989d53c57da15ba904837019ac8bf9c4f1f63656f8d26
Sep 17 18:09:45.917647 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:45.917607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" event={"ID":"b19561a3-053d-4ad5-9503-4650530db8a4","Type":"ContainerStarted","Data":"3919aed3296e4e816d6989d53c57da15ba904837019ac8bf9c4f1f63656f8d26"}
Sep 17 18:09:46.923368 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:46.923332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" event={"ID":"b19561a3-053d-4ad5-9503-4650530db8a4","Type":"ContainerStarted","Data":"79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d"}
Sep 17 18:09:46.923883 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:46.923481    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:09:46.944345 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:09:46.944298    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podStartSLOduration=1.944284283 podStartE2EDuration="1.944284283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:46.943147182 +0000 UTC m=+2111.903032444" watchObservedRunningTime="2026-09-17 18:09:46.944284283 +0000 UTC m=+2111.904169570"
Sep 17 18:10:17.930081 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:10:17.930049    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:11:52.684358 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:11:52.684279    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"]
Sep 17 18:11:57.930404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:11:57.930366    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:07.709771 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:07.709732    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main" containerID="cri-o://79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d" gracePeriod=45
Sep 17 18:12:07.929752 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:07.929713    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:17.930517 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:17.930466    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:17.931107 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:17.930619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:12:27.930089 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:27.930051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:37.953331 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:37.953309    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:12:38.086504 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.086469    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19561a3-053d-4ad5-9503-4650530db8a4-tls-certs\") pod \"b19561a3-053d-4ad5-9503-4650530db8a4\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") "
Sep 17 18:12:38.086709 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.086517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p57kp\" (UniqueName: \"kubernetes.io/projected/b19561a3-053d-4ad5-9503-4650530db8a4-kube-api-access-p57kp\") pod \"b19561a3-053d-4ad5-9503-4650530db8a4\" (UID: \"b19561a3-053d-4ad5-9503-4650530db8a4\") "
Sep 17 18:12:38.088760 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.088726    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19561a3-053d-4ad5-9503-4650530db8a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b19561a3-053d-4ad5-9503-4650530db8a4" (UID: "b19561a3-053d-4ad5-9503-4650530db8a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:38.088881 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.088733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19561a3-053d-4ad5-9503-4650530db8a4-kube-api-access-p57kp" (OuterVolumeSpecName: "kube-api-access-p57kp") pod "b19561a3-053d-4ad5-9503-4650530db8a4" (UID: "b19561a3-053d-4ad5-9503-4650530db8a4"). InnerVolumeSpecName "kube-api-access-p57kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:38.187151 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.187122    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19561a3-053d-4ad5-9503-4650530db8a4-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:12:38.187151 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.187147    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57kp\" (UniqueName: \"kubernetes.io/projected/b19561a3-053d-4ad5-9503-4650530db8a4-kube-api-access-p57kp\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:12:38.684041 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.684012    2561 generic.go:356] "Generic (PLEG): container finished" podID="b19561a3-053d-4ad5-9503-4650530db8a4" containerID="79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d" exitCode=0
Sep 17 18:12:38.684219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.684058    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"
Sep 17 18:12:38.684219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.684105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" event={"ID":"b19561a3-053d-4ad5-9503-4650530db8a4","Type":"ContainerDied","Data":"79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d"}
Sep 17 18:12:38.684219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.684154    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" event={"ID":"b19561a3-053d-4ad5-9503-4650530db8a4","Type":"ContainerDied","Data":"3919aed3296e4e816d6989d53c57da15ba904837019ac8bf9c4f1f63656f8d26"}
Sep 17 18:12:38.684219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.684177    2561 scope.go:117] "RemoveContainer" containerID="79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d"
Sep 17 18:12:38.693007 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.692987    2561 scope.go:117] "RemoveContainer" containerID="79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d"
Sep 17 18:12:38.693273 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:12:38.693257    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d\": container with ID starting with 79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d not found: ID does not exist" containerID="79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d"
Sep 17 18:12:38.693322 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.693280    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d"} err="failed to get container status \"79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d\": rpc error: code = NotFound desc = could not find container \"79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d\": container with ID starting with 79ca662f3654869c753200fb3bec896c5b0ee52310529e307112bfd7b34c917d not found: ID does not exist"
Sep 17 18:12:38.707100 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.707078    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"]
Sep 17 18:12:38.711104 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.711073    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg"]
Sep 17 18:12:38.929962 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:38.929909    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-s8mrg" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.41:9003\" within 1s: context deadline exceeded"
Sep 17 18:12:38.930129 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:12:38.930027    2561 logging.go:55] [core] [Channel #377 SubChannel #378]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.41:9003", ServerName: "10.132.0.41:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.41:9003: operation was canceled"
Sep 17 18:12:39.019616 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:39.019580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" path="/var/lib/kubelet/pods/b19561a3-053d-4ad5-9503-4650530db8a4/volumes"
Sep 17 18:12:43.151418 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.151386    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"]
Sep 17 18:12:43.151786 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.151705    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main"
Sep 17 18:12:43.151786 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.151715    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main"
Sep 17 18:12:43.151786 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.151785    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19561a3-053d-4ad5-9503-4650530db8a4" containerName="main"
Sep 17 18:12:43.156093 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.156077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.160709 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.160691    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:12:43.160941 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.160917    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:12:43.161037 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.161021    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-lst9x\""
Sep 17 18:12:43.161940 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.161919    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:12:43.185554 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.185529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"]
Sep 17 18:12:43.332488 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332488 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49ql\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-kube-api-access-k49ql\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ee4a0-2762-4275-b787-74747d2dc1da-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.332955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.332808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ee4a0-2762-4275-b787-74747d2dc1da-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433498    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ee4a0-2762-4275-b787-74747d2dc1da-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49ql\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-kube-api-access-k49ql\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.433866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ee4a0-2762-4275-b787-74747d2dc1da-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.434187 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.433970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.434187 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.434046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.434187 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.434089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.434341 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.434252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.434378 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.434345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ee4a0-2762-4275-b787-74747d2dc1da-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.435810 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.435785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.436196 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.436174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ee4a0-2762-4275-b787-74747d2dc1da-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.445884 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.445870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.451785 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.451764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49ql\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-kube-api-access-k49ql\") pod \"router-gateway-1-openshift-default-8555f5d76c-5jfnd\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.466202 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.466184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:43.741981 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.741959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"]
Sep 17 18:12:43.743858 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:12:43.743833    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952ee4a0_2762_4275_b787_74747d2dc1da.slice/crio-3803639d54fa6e83dd387decc57ff8618f8b82ad72211a1eb5a0d4621b871ad1 WatchSource:0}: Error finding container 3803639d54fa6e83dd387decc57ff8618f8b82ad72211a1eb5a0d4621b871ad1: Status 404 returned error can't find the container with id 3803639d54fa6e83dd387decc57ff8618f8b82ad72211a1eb5a0d4621b871ad1
Sep 17 18:12:43.745758 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.745730    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:12:43.745817 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.745791    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:12:43.745852 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:43.745818    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:12:44.714315 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:44.714279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" event={"ID":"952ee4a0-2762-4275-b787-74747d2dc1da","Type":"ContainerStarted","Data":"d2838c3304d568dbbefad685090f2475d150153e8d07221b13b949ae92b6bc4a"}
Sep 17 18:12:44.714834 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:44.714332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" event={"ID":"952ee4a0-2762-4275-b787-74747d2dc1da","Type":"ContainerStarted","Data":"3803639d54fa6e83dd387decc57ff8618f8b82ad72211a1eb5a0d4621b871ad1"}
Sep 17 18:12:44.737221 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:44.737174    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podStartSLOduration=1.737159996 podStartE2EDuration="1.737159996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:44.7359965 +0000 UTC m=+2289.695881770" watchObservedRunningTime="2026-09-17 18:12:44.737159996 +0000 UTC m=+2289.697045325"
Sep 17 18:12:45.466944 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:45.466908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:45.471732 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:45.471709    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:45.718827 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:45.718754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:45.719772 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:45.719754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:12:55.814841 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.814805    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"]
Sep 17 18:12:55.818258 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.818242    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.824552 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.824528    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-m748j\""
Sep 17 18:12:55.824715 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.824532    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:12:55.824715 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.824626    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:12:55.824715 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.824651    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:12:55.831716 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.831687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.831835 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.831735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44016359-358c-457f-b8fb-e331c66329cf-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.831909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.831830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.831909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.831872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrxg\" (UniqueName: \"kubernetes.io/projected/44016359-358c-457f-b8fb-e331c66329cf-kube-api-access-rxrxg\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.832017 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.831923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.832017 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.831945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.833659 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.833620    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"]
Sep 17 18:12:55.932907 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.932874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.932907 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.932912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrxg\" (UniqueName: \"kubernetes.io/projected/44016359-358c-457f-b8fb-e331c66329cf-kube-api-access-rxrxg\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.932936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.932951    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.933022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.933055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44016359-358c-457f-b8fb-e331c66329cf-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933332 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.933265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933389 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.933351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.933466 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.933444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.935162 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.935136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.935455 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.935437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44016359-358c-457f-b8fb-e331c66329cf-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:55.940424 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:55.940403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrxg\" (UniqueName: \"kubernetes.io/projected/44016359-358c-457f-b8fb-e331c66329cf-kube-api-access-rxrxg\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:56.128554 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:56.128471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:12:56.408855 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:56.408766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"]
Sep 17 18:12:56.412989 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:12:56.412957    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44016359_358c_457f_b8fb_e331c66329cf.slice/crio-afee0d48e73dd0bb83d7aebe1dc0f0c5da69f0e0571e72f837685cd009c51398 WatchSource:0}: Error finding container afee0d48e73dd0bb83d7aebe1dc0f0c5da69f0e0571e72f837685cd009c51398: Status 404 returned error can't find the container with id afee0d48e73dd0bb83d7aebe1dc0f0c5da69f0e0571e72f837685cd009c51398
Sep 17 18:12:56.771053 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:56.771007    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" event={"ID":"44016359-358c-457f-b8fb-e331c66329cf","Type":"ContainerStarted","Data":"ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"}
Sep 17 18:12:56.771263 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:56.771083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" event={"ID":"44016359-358c-457f-b8fb-e331c66329cf","Type":"ContainerStarted","Data":"afee0d48e73dd0bb83d7aebe1dc0f0c5da69f0e0571e72f837685cd009c51398"}
Sep 17 18:12:56.791603 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:56.791549    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" podStartSLOduration=1.79153364 podStartE2EDuration="1.79153364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:56.789890662 +0000 UTC m=+2301.749775931" watchObservedRunningTime="2026-09-17 18:12:56.79153364 +0000 UTC m=+2301.751418910"
Sep 17 18:12:57.129205 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:12:57.129125    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:13:06.129292 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:06.129259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:13:06.141745 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:06.141717    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:13:06.825219 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:06.825189    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:13:39.971079 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:39.971045    2561 generic.go:356] "Generic (PLEG): container finished" podID="952ee4a0-2762-4275-b787-74747d2dc1da" containerID="d2838c3304d568dbbefad685090f2475d150153e8d07221b13b949ae92b6bc4a" exitCode=0
Sep 17 18:13:39.971486 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:39.971119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" event={"ID":"952ee4a0-2762-4275-b787-74747d2dc1da","Type":"ContainerDied","Data":"d2838c3304d568dbbefad685090f2475d150153e8d07221b13b949ae92b6bc4a"}
Sep 17 18:13:39.971531 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:39.971519    2561 scope.go:117] "RemoveContainer" containerID="d2838c3304d568dbbefad685090f2475d150153e8d07221b13b949ae92b6bc4a"
Sep 17 18:13:39.972452 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:39.972424    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:13:39.972512 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:39.972490    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:13:39.972554 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:39.972521    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:13:40.978287 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:40.978254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" event={"ID":"952ee4a0-2762-4275-b787-74747d2dc1da","Type":"ContainerStarted","Data":"9f944994dede06b8687b05e97f0ca114120cd7ce63f068b4d2765bb41bd00a6f"}
Sep 17 18:13:41.466432 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:41.466400    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:13:41.467618 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:41.467589    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 17 18:13:42.467333 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:42.467289    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 17 18:13:43.467523 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:43.467482    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 17 18:13:44.467133 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:44.467097    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 17 18:13:45.467696 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:45.467650    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Sep 17 18:13:45.719735 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:45.719618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:13:46.470240 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:46.470212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:13:46.471271 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:46.471252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:13:51.591855 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.591819    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"]
Sep 17 18:13:51.595362 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.595339    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.598931 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.598906    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-9z6g7\""
Sep 17 18:13:51.599075 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.598966    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:13:51.605765 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.605740    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"]
Sep 17 18:13:51.634151 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.634106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/108e822f-7203-4e05-bc6b-a21abf22ec1b-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.634334 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.634182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8p87\" (UniqueName: \"kubernetes.io/projected/108e822f-7203-4e05-bc6b-a21abf22ec1b-kube-api-access-c8p87\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.734644 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.734598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/108e822f-7203-4e05-bc6b-a21abf22ec1b-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.734818 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.734697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8p87\" (UniqueName: \"kubernetes.io/projected/108e822f-7203-4e05-bc6b-a21abf22ec1b-kube-api-access-c8p87\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.737146 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.737125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/108e822f-7203-4e05-bc6b-a21abf22ec1b-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.743776 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.743749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8p87\" (UniqueName: \"kubernetes.io/projected/108e822f-7203-4e05-bc6b-a21abf22ec1b-kube-api-access-c8p87\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:51.906597 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:51.906500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:52.181755 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:52.181669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"]
Sep 17 18:13:52.184978 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:13:52.184951    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108e822f_7203_4e05_bc6b_a21abf22ec1b.slice/crio-e92bf3005699334512f80c4ea5aa03bd56e30271098652c720b9593a75c7f101 WatchSource:0}: Error finding container e92bf3005699334512f80c4ea5aa03bd56e30271098652c720b9593a75c7f101: Status 404 returned error can't find the container with id e92bf3005699334512f80c4ea5aa03bd56e30271098652c720b9593a75c7f101
Sep 17 18:13:53.038167 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:53.038127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" event={"ID":"108e822f-7203-4e05-bc6b-a21abf22ec1b","Type":"ContainerStarted","Data":"41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca"}
Sep 17 18:13:53.038609 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:53.038180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" event={"ID":"108e822f-7203-4e05-bc6b-a21abf22ec1b","Type":"ContainerStarted","Data":"e92bf3005699334512f80c4ea5aa03bd56e30271098652c720b9593a75c7f101"}
Sep 17 18:13:53.038609 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:53.038220    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:13:53.058338 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:13:53.058286    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" podStartSLOduration=2.058268405 podStartE2EDuration="2.058268405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:53.056209962 +0000 UTC m=+2358.016095246" watchObservedRunningTime="2026-09-17 18:13:53.058268405 +0000 UTC m=+2358.018153676"
Sep 17 18:14:02.951864 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:02.951826    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"]
Sep 17 18:14:02.952288 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:02.952063    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy" containerID="cri-o://9f944994dede06b8687b05e97f0ca114120cd7ce63f068b4d2765bb41bd00a6f" gracePeriod=30
Sep 17 18:14:08.109592 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.109548    2561 generic.go:356] "Generic (PLEG): container finished" podID="952ee4a0-2762-4275-b787-74747d2dc1da" containerID="9f944994dede06b8687b05e97f0ca114120cd7ce63f068b4d2765bb41bd00a6f" exitCode=0
Sep 17 18:14:08.110032 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.109607    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" event={"ID":"952ee4a0-2762-4275-b787-74747d2dc1da","Type":"ContainerDied","Data":"9f944994dede06b8687b05e97f0ca114120cd7ce63f068b4d2765bb41bd00a6f"}
Sep 17 18:14:08.110032 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.109682    2561 scope.go:117] "RemoveContainer" containerID="d2838c3304d568dbbefad685090f2475d150153e8d07221b13b949ae92b6bc4a"
Sep 17 18:14:08.198433 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.198409    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:14:08.274521 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-certs\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274719 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274559    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-credential-socket\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274719 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k49ql\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-kube-api-access-k49ql\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274719 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274654    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-socket\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274719 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ee4a0-2762-4275-b787-74747d2dc1da-istiod-ca-cert\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ee4a0-2762-4275-b787-74747d2dc1da-istio-podinfo\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274783    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-data\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-envoy\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.274955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274827    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:08.274955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274838    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:08.274955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.274855    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-istio-token\") pod \"952ee4a0-2762-4275-b787-74747d2dc1da\" (UID: \"952ee4a0-2762-4275-b787-74747d2dc1da\") "
Sep 17 18:14:08.275261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.275028    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-data" (OuterVolumeSpecName: "istio-data") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:08.275261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.275199    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-credential-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.275261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.275216    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-data\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.275261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.275229    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.275261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.275233    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:08.275528 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.275292    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952ee4a0-2762-4275-b787-74747d2dc1da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:08.277329 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.277301    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/952ee4a0-2762-4275-b787-74747d2dc1da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:14:08.277521 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.277499    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-kube-api-access-k49ql" (OuterVolumeSpecName: "kube-api-access-k49ql") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "kube-api-access-k49ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:08.277624 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.277591    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:08.277624 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.277522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-istio-token" (OuterVolumeSpecName: "istio-token") pod "952ee4a0-2762-4275-b787-74747d2dc1da" (UID: "952ee4a0-2762-4275-b787-74747d2dc1da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:08.376356 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.376319    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-istio-envoy\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.376356 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.376347    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-istio-token\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.376356 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.376357    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k49ql\" (UniqueName: \"kubernetes.io/projected/952ee4a0-2762-4275-b787-74747d2dc1da-kube-api-access-k49ql\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.376593 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.376367    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/952ee4a0-2762-4275-b787-74747d2dc1da-workload-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.376593 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.376376    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/952ee4a0-2762-4275-b787-74747d2dc1da-istiod-ca-cert\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:08.376593 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:08.376383    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/952ee4a0-2762-4275-b787-74747d2dc1da-istio-podinfo\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:14:09.116038 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:09.116004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd" event={"ID":"952ee4a0-2762-4275-b787-74747d2dc1da","Type":"ContainerDied","Data":"3803639d54fa6e83dd387decc57ff8618f8b82ad72211a1eb5a0d4621b871ad1"}
Sep 17 18:14:09.116462 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:09.116059    2561 scope.go:117] "RemoveContainer" containerID="9f944994dede06b8687b05e97f0ca114120cd7ce63f068b4d2765bb41bd00a6f"
Sep 17 18:14:09.116462 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:09.116014    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"
Sep 17 18:14:09.135557 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:09.135530    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"]
Sep 17 18:14:09.139519 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:09.139497    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-5jfnd"]
Sep 17 18:14:11.020159 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:11.020127    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" path="/var/lib/kubelet/pods/952ee4a0-2762-4275-b787-74747d2dc1da/volumes"
Sep 17 18:14:19.315133 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315098    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"]
Sep 17 18:14:19.315668 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315650    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy"
Sep 17 18:14:19.315759 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315670    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy"
Sep 17 18:14:19.315759 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy"
Sep 17 18:14:19.315759 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315728    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy"
Sep 17 18:14:19.315911 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315811    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy"
Sep 17 18:14:19.315911 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.315827    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="952ee4a0-2762-4275-b787-74747d2dc1da" containerName="istio-proxy"
Sep 17 18:14:19.320029 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.320009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.322617 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.322579    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:14:19.323852 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.323830    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:14:19.323852 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.323841    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:14:19.324015 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.323862    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-lrd2b\""
Sep 17 18:14:19.324015 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.323863    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-rrl2w\""
Sep 17 18:14:19.329520 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.329497    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"]
Sep 17 18:14:19.474857 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.474819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a1786-edbd-4d46-a3be-b15dffa9ac3f-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.475042 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.474903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrsx\" (UniqueName: \"kubernetes.io/projected/110a1786-edbd-4d46-a3be-b15dffa9ac3f-kube-api-access-rwrsx\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.575465 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.575386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrsx\" (UniqueName: \"kubernetes.io/projected/110a1786-edbd-4d46-a3be-b15dffa9ac3f-kube-api-access-rwrsx\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.575613 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.575474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a1786-edbd-4d46-a3be-b15dffa9ac3f-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.578025 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.578002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a1786-edbd-4d46-a3be-b15dffa9ac3f-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.583722 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.583701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrsx\" (UniqueName: \"kubernetes.io/projected/110a1786-edbd-4d46-a3be-b15dffa9ac3f-kube-api-access-rwrsx\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.631332 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.631303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:19.906468 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:19.906382    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"]
Sep 17 18:14:19.909816 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:14:19.909786    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110a1786_edbd_4d46_a3be_b15dffa9ac3f.slice/crio-5a042513ec276381e50fbb8db1d5b0ba968af271ce5d4a2476593db861dc915d WatchSource:0}: Error finding container 5a042513ec276381e50fbb8db1d5b0ba968af271ce5d4a2476593db861dc915d: Status 404 returned error can't find the container with id 5a042513ec276381e50fbb8db1d5b0ba968af271ce5d4a2476593db861dc915d
Sep 17 18:14:20.168067 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:20.167988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" event={"ID":"110a1786-edbd-4d46-a3be-b15dffa9ac3f","Type":"ContainerStarted","Data":"a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239"}
Sep 17 18:14:20.168067 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:20.168037    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" event={"ID":"110a1786-edbd-4d46-a3be-b15dffa9ac3f","Type":"ContainerStarted","Data":"5a042513ec276381e50fbb8db1d5b0ba968af271ce5d4a2476593db861dc915d"}
Sep 17 18:14:20.168245 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:20.168085    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:20.186260 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:20.186218    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" podStartSLOduration=1.186205707 podStartE2EDuration="1.186205707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:20.184616752 +0000 UTC m=+2385.144502020" watchObservedRunningTime="2026-09-17 18:14:20.186205707 +0000 UTC m=+2385.146090976"
Sep 17 18:14:24.044930 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:24.044896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:14:36.562442 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:36.562411    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"]
Sep 17 18:14:37.121592 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:37.121564    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:14:37.125950 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:37.125924    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:14:44.044055 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:44.044010    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:51.175179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:51.175104    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:14:51.583283 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:51.583217    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main" containerID="cri-o://41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca" gracePeriod=45
Sep 17 18:14:54.044246 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:14:54.044210    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:04.044488 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:04.044453    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:04.044889 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:04.044547    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:15:14.044526 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:14.044484    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:21.835990 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:21.835966    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:15:21.918972 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:21.918885    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8p87\" (UniqueName: \"kubernetes.io/projected/108e822f-7203-4e05-bc6b-a21abf22ec1b-kube-api-access-c8p87\") pod \"108e822f-7203-4e05-bc6b-a21abf22ec1b\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") "
Sep 17 18:15:21.919141 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:21.918974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/108e822f-7203-4e05-bc6b-a21abf22ec1b-tls-certs\") pod \"108e822f-7203-4e05-bc6b-a21abf22ec1b\" (UID: \"108e822f-7203-4e05-bc6b-a21abf22ec1b\") "
Sep 17 18:15:21.921116 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:21.921078    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108e822f-7203-4e05-bc6b-a21abf22ec1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "108e822f-7203-4e05-bc6b-a21abf22ec1b" (UID: "108e822f-7203-4e05-bc6b-a21abf22ec1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:21.921222 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:21.921140    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108e822f-7203-4e05-bc6b-a21abf22ec1b-kube-api-access-c8p87" (OuterVolumeSpecName: "kube-api-access-c8p87") pod "108e822f-7203-4e05-bc6b-a21abf22ec1b" (UID: "108e822f-7203-4e05-bc6b-a21abf22ec1b"). InnerVolumeSpecName "kube-api-access-c8p87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:22.020551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.020517    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/108e822f-7203-4e05-bc6b-a21abf22ec1b-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.020551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.020546    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8p87\" (UniqueName: \"kubernetes.io/projected/108e822f-7203-4e05-bc6b-a21abf22ec1b-kube-api-access-c8p87\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:22.457986 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.457952    2561 generic.go:356] "Generic (PLEG): container finished" podID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerID="41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca" exitCode=0
Sep 17 18:15:22.458174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.457990    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"
Sep 17 18:15:22.458174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.458035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" event={"ID":"108e822f-7203-4e05-bc6b-a21abf22ec1b","Type":"ContainerDied","Data":"41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca"}
Sep 17 18:15:22.458174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.458085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z" event={"ID":"108e822f-7203-4e05-bc6b-a21abf22ec1b","Type":"ContainerDied","Data":"e92bf3005699334512f80c4ea5aa03bd56e30271098652c720b9593a75c7f101"}
Sep 17 18:15:22.458174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.458114    2561 scope.go:117] "RemoveContainer" containerID="41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca"
Sep 17 18:15:22.467315 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.467297    2561 scope.go:117] "RemoveContainer" containerID="41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca"
Sep 17 18:15:22.467579 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:15:22.467563    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca\": container with ID starting with 41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca not found: ID does not exist" containerID="41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca"
Sep 17 18:15:22.467625 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.467587    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca"} err="failed to get container status \"41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca\": rpc error: code = NotFound desc = could not find container \"41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca\": container with ID starting with 41c349a6143911740281927b801c291e04cad09cc37e6f8c0249a0e3c828e9ca not found: ID does not exist"
Sep 17 18:15:22.481627 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.481602    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"]
Sep 17 18:15:22.487104 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:22.487079    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche56m6z"]
Sep 17 18:15:23.019772 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:23.019738    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" path="/var/lib/kubelet/pods/108e822f-7203-4e05-bc6b-a21abf22ec1b/volumes"
Sep 17 18:15:25.327619 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.327586    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"]
Sep 17 18:15:25.336565 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:15:25.336529    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:15:25.336565 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:25.336565 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:25.336565 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4" cmd=["/bin/sleep","15"]
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.336587    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" message="" err=<
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:15:25.336604    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" podUID="44016359-358c-457f-b8fb-e331c66329cf" containerName="main" containerID="cri-o://ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"
Sep 17 18:15:25.336895 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.336673    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" podUID="44016359-358c-457f-b8fb-e331c66329cf" containerName="main" containerID="cri-o://ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4" gracePeriod=60
Sep 17 18:15:25.695118 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.695093    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:15:25.856835 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.856742    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-dshm\") pod \"44016359-358c-457f-b8fb-e331c66329cf\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") "
Sep 17 18:15:25.856835 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.856792    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-model-cache\") pod \"44016359-358c-457f-b8fb-e331c66329cf\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") "
Sep 17 18:15:25.857047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.856848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-home\") pod \"44016359-358c-457f-b8fb-e331c66329cf\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") "
Sep 17 18:15:25.857047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.856880    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxrxg\" (UniqueName: \"kubernetes.io/projected/44016359-358c-457f-b8fb-e331c66329cf-kube-api-access-rxrxg\") pod \"44016359-358c-457f-b8fb-e331c66329cf\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") "
Sep 17 18:15:25.857047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.856936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44016359-358c-457f-b8fb-e331c66329cf-tls-certs\") pod \"44016359-358c-457f-b8fb-e331c66329cf\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") "
Sep 17 18:15:25.857047 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.856974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-tmp-dir\") pod \"44016359-358c-457f-b8fb-e331c66329cf\" (UID: \"44016359-358c-457f-b8fb-e331c66329cf\") "
Sep 17 18:15:25.857274 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.857096    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-home" (OuterVolumeSpecName: "home") pod "44016359-358c-457f-b8fb-e331c66329cf" (UID: "44016359-358c-457f-b8fb-e331c66329cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:25.857274 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.857141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "44016359-358c-457f-b8fb-e331c66329cf" (UID: "44016359-358c-457f-b8fb-e331c66329cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:25.857274 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.857253    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44016359-358c-457f-b8fb-e331c66329cf" (UID: "44016359-358c-457f-b8fb-e331c66329cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:25.857398 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.857309    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:25.857398 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.857327    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:25.859022 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.858994    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-dshm" (OuterVolumeSpecName: "dshm") pod "44016359-358c-457f-b8fb-e331c66329cf" (UID: "44016359-358c-457f-b8fb-e331c66329cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:25.859136 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.859038    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44016359-358c-457f-b8fb-e331c66329cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44016359-358c-457f-b8fb-e331c66329cf" (UID: "44016359-358c-457f-b8fb-e331c66329cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:25.859136 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.859054    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44016359-358c-457f-b8fb-e331c66329cf-kube-api-access-rxrxg" (OuterVolumeSpecName: "kube-api-access-rxrxg") pod "44016359-358c-457f-b8fb-e331c66329cf" (UID: "44016359-358c-457f-b8fb-e331c66329cf"). InnerVolumeSpecName "kube-api-access-rxrxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:25.958725 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.958678    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrxg\" (UniqueName: \"kubernetes.io/projected/44016359-358c-457f-b8fb-e331c66329cf-kube-api-access-rxrxg\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:25.958725 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.958717    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44016359-358c-457f-b8fb-e331c66329cf-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:25.958725 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.958727    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:25.958725 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:25.958735    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44016359-358c-457f-b8fb-e331c66329cf-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:15:26.478751 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.478708    2561 generic.go:356] "Generic (PLEG): container finished" podID="44016359-358c-457f-b8fb-e331c66329cf" containerID="ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4" exitCode=0
Sep 17 18:15:26.479233 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.478772    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"
Sep 17 18:15:26.479233 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.478793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" event={"ID":"44016359-358c-457f-b8fb-e331c66329cf","Type":"ContainerDied","Data":"ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"}
Sep 17 18:15:26.479233 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.478890    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww" event={"ID":"44016359-358c-457f-b8fb-e331c66329cf","Type":"ContainerDied","Data":"afee0d48e73dd0bb83d7aebe1dc0f0c5da69f0e0571e72f837685cd009c51398"}
Sep 17 18:15:26.479233 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.478924    2561 scope.go:117] "RemoveContainer" containerID="ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"
Sep 17 18:15:26.489875 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.489849    2561 scope.go:117] "RemoveContainer" containerID="ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"
Sep 17 18:15:26.490288 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:15:26.490267    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4\": container with ID starting with ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4 not found: ID does not exist" containerID="ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"
Sep 17 18:15:26.490380 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.490295    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4"} err="failed to get container status \"ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4\": rpc error: code = NotFound desc = could not find container \"ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4\": container with ID starting with ffaa042ddc179ea85b8afc8ec19ec2d48e39983a52eee735dd621b7bf3adfab4 not found: ID does not exist"
Sep 17 18:15:26.500469 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.500447    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"]
Sep 17 18:15:26.503860 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:26.503839    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fb5ww"]
Sep 17 18:15:27.020051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:15:27.020020    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44016359-358c-457f-b8fb-e331c66329cf" path="/var/lib/kubelet/pods/44016359-358c-457f-b8fb-e331c66329cf/volumes"
Sep 17 18:16:20.083899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.083822    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"]
Sep 17 18:16:20.084367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.084191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main"
Sep 17 18:16:20.084367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.084205    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main"
Sep 17 18:16:20.084367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.084224    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44016359-358c-457f-b8fb-e331c66329cf" containerName="main"
Sep 17 18:16:20.084367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.084229    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44016359-358c-457f-b8fb-e331c66329cf" containerName="main"
Sep 17 18:16:20.084367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.084288    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="108e822f-7203-4e05-bc6b-a21abf22ec1b" containerName="main"
Sep 17 18:16:20.084367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.084298    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="44016359-358c-457f-b8fb-e331c66329cf" containerName="main"
Sep 17 18:16:20.087369 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.087353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.089937 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.089918    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:16:20.090930 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.090909    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:16:20.091071 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.090984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-pg56v\""
Sep 17 18:16:20.091071 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.090984    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:16:20.096619 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.096597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"]
Sep 17 18:16:20.117909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.117885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.118031 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.117957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.118031 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.118005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.118134 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.118046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b218dc-5da2-4d6b-954f-2e6d95436307-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.118134 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.118082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcqlh\" (UniqueName: \"kubernetes.io/projected/23b218dc-5da2-4d6b-954f-2e6d95436307-kube-api-access-lcqlh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.118214 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.118126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219215 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcqlh\" (UniqueName: \"kubernetes.io/projected/23b218dc-5da2-4d6b-954f-2e6d95436307-kube-api-access-lcqlh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219388 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219388 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219388 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219323    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219388 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219664 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b218dc-5da2-4d6b-954f-2e6d95436307-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219737 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219798 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219733    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.219798 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.219780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.221694 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.221673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.221886 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.221869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b218dc-5da2-4d6b-954f-2e6d95436307-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.228773 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.228748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcqlh\" (UniqueName: \"kubernetes.io/projected/23b218dc-5da2-4d6b-954f-2e6d95436307-kube-api-access-lcqlh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.398731 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.398641    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:20.680863 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.680803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"]
Sep 17 18:16:20.683405 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:16:20.683377    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b218dc_5da2_4d6b_954f_2e6d95436307.slice/crio-ed1c4e7eacb44d29e78f21bc86ddb5c2b949fc13e1f9e21d396043939df32f36 WatchSource:0}: Error finding container ed1c4e7eacb44d29e78f21bc86ddb5c2b949fc13e1f9e21d396043939df32f36: Status 404 returned error can't find the container with id ed1c4e7eacb44d29e78f21bc86ddb5c2b949fc13e1f9e21d396043939df32f36
Sep 17 18:16:20.724530 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:20.724501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" event={"ID":"23b218dc-5da2-4d6b-954f-2e6d95436307","Type":"ContainerStarted","Data":"ed1c4e7eacb44d29e78f21bc86ddb5c2b949fc13e1f9e21d396043939df32f36"}
Sep 17 18:16:21.730369 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:21.730335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" event={"ID":"23b218dc-5da2-4d6b-954f-2e6d95436307","Type":"ContainerStarted","Data":"3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"}
Sep 17 18:16:21.750318 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:21.750270    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" podStartSLOduration=1.7502546780000001 podStartE2EDuration="1.750254678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:21.749202858 +0000 UTC m=+2506.709088129" watchObservedRunningTime="2026-09-17 18:16:21.750254678 +0000 UTC m=+2506.710139999"
Sep 17 18:16:22.399274 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:22.399240    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:30.399071 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:30.399027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:30.411477 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:30.411446    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:30.782132 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:30.782102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:16:32.508063 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:32.508027    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"]
Sep 17 18:16:41.174300 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:41.174261    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:16:47.527359 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:47.527301    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main" containerID="cri-o://a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239" gracePeriod=45
Sep 17 18:16:51.174252 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:16:51.174209    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:01.175037 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:01.174997    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:01.175469 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:01.175111    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:17:11.174561 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:11.174523    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:17.774719 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.774689    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:17:17.823168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.823100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwrsx\" (UniqueName: \"kubernetes.io/projected/110a1786-edbd-4d46-a3be-b15dffa9ac3f-kube-api-access-rwrsx\") pod \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") "
Sep 17 18:17:17.823168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.823143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a1786-edbd-4d46-a3be-b15dffa9ac3f-tls-certs\") pod \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\" (UID: \"110a1786-edbd-4d46-a3be-b15dffa9ac3f\") "
Sep 17 18:17:17.825287 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.825257    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110a1786-edbd-4d46-a3be-b15dffa9ac3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "110a1786-edbd-4d46-a3be-b15dffa9ac3f" (UID: "110a1786-edbd-4d46-a3be-b15dffa9ac3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:17.825391 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.825363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110a1786-edbd-4d46-a3be-b15dffa9ac3f-kube-api-access-rwrsx" (OuterVolumeSpecName: "kube-api-access-rwrsx") pod "110a1786-edbd-4d46-a3be-b15dffa9ac3f" (UID: "110a1786-edbd-4d46-a3be-b15dffa9ac3f"). InnerVolumeSpecName "kube-api-access-rwrsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:17.923896 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.923864    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrsx\" (UniqueName: \"kubernetes.io/projected/110a1786-edbd-4d46-a3be-b15dffa9ac3f-kube-api-access-rwrsx\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:17:17.923896 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.923890    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110a1786-edbd-4d46-a3be-b15dffa9ac3f-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:17:17.987829 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.987798    2561 generic.go:356] "Generic (PLEG): container finished" podID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerID="a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239" exitCode=0
Sep 17 18:17:17.987985 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.987836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" event={"ID":"110a1786-edbd-4d46-a3be-b15dffa9ac3f","Type":"ContainerDied","Data":"a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239"}
Sep 17 18:17:17.987985 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.987845    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"
Sep 17 18:17:17.987985 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.987887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p" event={"ID":"110a1786-edbd-4d46-a3be-b15dffa9ac3f","Type":"ContainerDied","Data":"5a042513ec276381e50fbb8db1d5b0ba968af271ce5d4a2476593db861dc915d"}
Sep 17 18:17:17.987985 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.987916    2561 scope.go:117] "RemoveContainer" containerID="a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239"
Sep 17 18:17:17.998420 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.998395    2561 scope.go:117] "RemoveContainer" containerID="a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239"
Sep 17 18:17:17.998724 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:17:17.998703    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239\": container with ID starting with a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239 not found: ID does not exist" containerID="a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239"
Sep 17 18:17:17.998799 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:17.998733    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239"} err="failed to get container status \"a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239\": rpc error: code = NotFound desc = could not find container \"a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239\": container with ID starting with a29128c4d1751738a55524e5e0b2d9eb5e3f8ff590978463db1b623c14fef239 not found: ID does not exist"
Sep 17 18:17:18.011563 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:18.011542    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"]
Sep 17 18:17:18.019985 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:18.019959    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schepgw8p"]
Sep 17 18:17:19.019464 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:19.019427    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" path="/var/lib/kubelet/pods/110a1786-edbd-4d46-a3be-b15dffa9ac3f/volumes"
Sep 17 18:17:31.014414 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.014381    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"]
Sep 17 18:17:31.014891 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.014795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main"
Sep 17 18:17:31.014891 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.014809    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main"
Sep 17 18:17:31.014891 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.014877    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="110a1786-edbd-4d46-a3be-b15dffa9ac3f" containerName="main"
Sep 17 18:17:31.018369 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.018349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.021110 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.021090    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-57ksd\""
Sep 17 18:17:31.021230 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.021125    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:17:31.021230 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.021164    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:17:31.021347 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.021233    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:17:31.030065 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.030039    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"]
Sep 17 18:17:31.134282 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.134448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.134448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.134448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.134448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.134448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.134448 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.134447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcrq\" (UniqueName: \"kubernetes.io/projected/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kube-api-access-qqcrq\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235147 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235147 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235350 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235350 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcrq\" (UniqueName: \"kubernetes.io/projected/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kube-api-access-qqcrq\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235439 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235398    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235525 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235584 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235665 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235733 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235853 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.235951 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.235623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.237769 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.237747    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.238009 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.237991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.244925 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.244899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcrq\" (UniqueName: \"kubernetes.io/projected/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kube-api-access-qqcrq\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.328912 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.328840    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:17:31.335518 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.335491    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"]
Sep 17 18:17:31.339434 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.339349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.341987 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.341963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-kncrx\""
Sep 17 18:17:31.347746 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.347714    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"]
Sep 17 18:17:31.436972 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.436941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e44e62-a05e-4f2a-a4a2-a6f639282857-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.437109 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.436997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgff\" (UniqueName: \"kubernetes.io/projected/01e44e62-a05e-4f2a-a4a2-a6f639282857-kube-api-access-jvgff\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.538459 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.538428    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgff\" (UniqueName: \"kubernetes.io/projected/01e44e62-a05e-4f2a-a4a2-a6f639282857-kube-api-access-jvgff\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.538648 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.538509    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e44e62-a05e-4f2a-a4a2-a6f639282857-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.541136 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.541110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e44e62-a05e-4f2a-a4a2-a6f639282857-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.546981 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.546954    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgff\" (UniqueName: \"kubernetes.io/projected/01e44e62-a05e-4f2a-a4a2-a6f639282857-kube-api-access-jvgff\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.609542 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.609517    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"]
Sep 17 18:17:31.653273 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.653248    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:17:31.665593 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.665572    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:31.939907 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:31.939881    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"]
Sep 17 18:17:31.941618 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:17:31.941595    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e44e62_a05e_4f2a_a4a2_a6f639282857.slice/crio-4b14508b87b0f3dabee1e88c6039195f94833851af30f24d846bd125011d2fa0 WatchSource:0}: Error finding container 4b14508b87b0f3dabee1e88c6039195f94833851af30f24d846bd125011d2fa0: Status 404 returned error can't find the container with id 4b14508b87b0f3dabee1e88c6039195f94833851af30f24d846bd125011d2fa0
Sep 17 18:17:32.049664 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:32.049602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" event={"ID":"01e44e62-a05e-4f2a-a4a2-a6f639282857","Type":"ContainerStarted","Data":"c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda"}
Sep 17 18:17:32.050138 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:32.049884    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:17:32.050138 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:32.049937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" event={"ID":"01e44e62-a05e-4f2a-a4a2-a6f639282857","Type":"ContainerStarted","Data":"4b14508b87b0f3dabee1e88c6039195f94833851af30f24d846bd125011d2fa0"}
Sep 17 18:17:32.051447 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:32.051411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerStarted","Data":"6e366804ce645f84e64363cd0ae343de8f03883c1c0aaf5bd3ea44bc00d82e49"}
Sep 17 18:17:32.084601 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:32.078521    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podStartSLOduration=1.078500118 podStartE2EDuration="1.078500118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:32.075150394 +0000 UTC m=+2577.035035666" watchObservedRunningTime="2026-09-17 18:17:32.078500118 +0000 UTC m=+2577.038385390"
Sep 17 18:17:40.092861 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:40.092827    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerStarted","Data":"e84785947ea9f70e581cb843e4a7a3166a44ff9f2baa42a66a675c03e8b7b59a"}
Sep 17 18:17:41.098186 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:41.098095    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerID="e84785947ea9f70e581cb843e4a7a3166a44ff9f2baa42a66a675c03e8b7b59a" exitCode=0
Sep 17 18:17:41.098675 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:17:41.098176    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerDied","Data":"e84785947ea9f70e581cb843e4a7a3166a44ff9f2baa42a66a675c03e8b7b59a"}
Sep 17 18:18:03.057229 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:03.057140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:18:11.272831 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:11.272797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerStarted","Data":"8b53dd4c94ef1746d47b881df830f9292118cd838fcc2c0bf7c67226425bca9c"}
Sep 17 18:18:11.273345 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:11.272888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerStarted","Data":"7974c185d76434b1c269ffbfefabbcb9b413d9c426c6aaaa9d017c0530e9b9c6"}
Sep 17 18:18:11.298306 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:11.298248    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podStartSLOduration=2.962173216 podStartE2EDuration="41.298226886s" totalImagesPullingTime="37.249642155s" totalInitContainerRuntime="1.086411515s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:11.296019736 +0000 UTC m=+2616.255905006" watchObservedRunningTime="2026-09-17 18:18:11.298226886 +0000 UTC m=+2616.258112156"
Sep 17 18:18:11.329316 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:11.329243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:18:11.329316 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:11.329301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:18:11.331054 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:11.331030    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:18:21.329716 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:21.329678    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:18:31.329270 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:31.329224    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:18:41.330213 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:41.330164    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:18:47.556617 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.556582    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"]
Sep 17 18:18:47.569045 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:18:47.569012    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:18:47.569045 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:47.569045 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:47.569045 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87" cmd=["/bin/sleep","15"]
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.569062    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" message="" err=<
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:18:47.569075    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerName="main" containerID="cri-o://3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"
Sep 17 18:18:47.569378 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.569110    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerName="main" containerID="cri-o://3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87" gracePeriod=60
Sep 17 18:18:47.771302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.771261    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 18:18:47.915361 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.915335    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:18:47.952222 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952188    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-model-cache\") pod \"23b218dc-5da2-4d6b-954f-2e6d95436307\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") "
Sep 17 18:18:47.952428 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952246    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-tmp-dir\") pod \"23b218dc-5da2-4d6b-954f-2e6d95436307\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") "
Sep 17 18:18:47.952428 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b218dc-5da2-4d6b-954f-2e6d95436307-tls-certs\") pod \"23b218dc-5da2-4d6b-954f-2e6d95436307\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") "
Sep 17 18:18:47.952428 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-dshm\") pod \"23b218dc-5da2-4d6b-954f-2e6d95436307\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") "
Sep 17 18:18:47.952428 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952385    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-home\") pod \"23b218dc-5da2-4d6b-954f-2e6d95436307\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") "
Sep 17 18:18:47.952624 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-model-cache" (OuterVolumeSpecName: "model-cache") pod "23b218dc-5da2-4d6b-954f-2e6d95436307" (UID: "23b218dc-5da2-4d6b-954f-2e6d95436307"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:47.952624 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23b218dc-5da2-4d6b-954f-2e6d95436307" (UID: "23b218dc-5da2-4d6b-954f-2e6d95436307"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:47.952624 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-home" (OuterVolumeSpecName: "home") pod "23b218dc-5da2-4d6b-954f-2e6d95436307" (UID: "23b218dc-5da2-4d6b-954f-2e6d95436307"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:47.952624 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcqlh\" (UniqueName: \"kubernetes.io/projected/23b218dc-5da2-4d6b-954f-2e6d95436307-kube-api-access-lcqlh\") pod \"23b218dc-5da2-4d6b-954f-2e6d95436307\" (UID: \"23b218dc-5da2-4d6b-954f-2e6d95436307\") "
Sep 17 18:18:47.952952 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952928    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:18:47.952952 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952953    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:18:47.953125 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.952967    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:18:47.955207 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.955179    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b218dc-5da2-4d6b-954f-2e6d95436307-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23b218dc-5da2-4d6b-954f-2e6d95436307" (UID: "23b218dc-5da2-4d6b-954f-2e6d95436307"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:47.955294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.955204    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-dshm" (OuterVolumeSpecName: "dshm") pod "23b218dc-5da2-4d6b-954f-2e6d95436307" (UID: "23b218dc-5da2-4d6b-954f-2e6d95436307"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:47.955294 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:47.955255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b218dc-5da2-4d6b-954f-2e6d95436307-kube-api-access-lcqlh" (OuterVolumeSpecName: "kube-api-access-lcqlh") pod "23b218dc-5da2-4d6b-954f-2e6d95436307" (UID: "23b218dc-5da2-4d6b-954f-2e6d95436307"). InnerVolumeSpecName "kube-api-access-lcqlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:48.053441 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.053405    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b218dc-5da2-4d6b-954f-2e6d95436307-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:18:48.053441 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.053433    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcqlh\" (UniqueName: \"kubernetes.io/projected/23b218dc-5da2-4d6b-954f-2e6d95436307-kube-api-access-lcqlh\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:18:48.053441 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.053444    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b218dc-5da2-4d6b-954f-2e6d95436307-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:18:48.465461 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.465430    2561 generic.go:356] "Generic (PLEG): container finished" podID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerID="3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87" exitCode=0
Sep 17 18:18:48.465657 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.465488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" event={"ID":"23b218dc-5da2-4d6b-954f-2e6d95436307","Type":"ContainerDied","Data":"3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"}
Sep 17 18:18:48.465657 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.465491    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"
Sep 17 18:18:48.465657 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.465528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz" event={"ID":"23b218dc-5da2-4d6b-954f-2e6d95436307","Type":"ContainerDied","Data":"ed1c4e7eacb44d29e78f21bc86ddb5c2b949fc13e1f9e21d396043939df32f36"}
Sep 17 18:18:48.465657 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.465550    2561 scope.go:117] "RemoveContainer" containerID="3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"
Sep 17 18:18:48.474932 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.474913    2561 scope.go:117] "RemoveContainer" containerID="3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"
Sep 17 18:18:48.475245 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:18:48.475227    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87\": container with ID starting with 3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87 not found: ID does not exist" containerID="3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"
Sep 17 18:18:48.475301 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.475256    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87"} err="failed to get container status \"3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87\": rpc error: code = NotFound desc = could not find container \"3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87\": container with ID starting with 3f1955163b15d16cb28bcfe992415ec4fc6c8d5741957e0055d2e5b3e1b2ad87 not found: ID does not exist"
Sep 17 18:18:48.490737 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.490708    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"]
Sep 17 18:18:48.494542 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:48.494516    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969ldldz"]
Sep 17 18:18:49.020919 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:49.020890    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" path="/var/lib/kubelet/pods/23b218dc-5da2-4d6b-954f-2e6d95436307/volumes"
Sep 17 18:18:51.329606 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:18:51.329566    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:19:01.329760 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:01.329718    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:19:11.329514 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:11.329466    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:19:21.329500 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:21.329417    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:19:31.329608 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:31.329569    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:19:37.150690 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:37.150663    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:19:37.155476 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:37.155452    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:19:41.330400 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:41.330342    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:19:51.329551 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:19:51.329503    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:20:01.339076 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:01.339047    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:20:01.346723 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:01.346704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:20:09.150520 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:09.150484    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"]
Sep 17 18:20:09.151084 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:09.150790    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="modelcar" containerID="cri-o://8b53dd4c94ef1746d47b881df830f9292118cd838fcc2c0bf7c67226425bca9c" gracePeriod=60
Sep 17 18:20:09.166153 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:09.166125    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"]
Sep 17 18:20:09.857743 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:09.857713    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4_9d346f08-7fcc-4a63-8db1-5fbf12a26965/modelcar/0.log"
Sep 17 18:20:09.858451 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:09.858428    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerID="8b53dd4c94ef1746d47b881df830f9292118cd838fcc2c0bf7c67226425bca9c" exitCode=143
Sep 17 18:20:09.858553 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:09.858505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerDied","Data":"8b53dd4c94ef1746d47b881df830f9292118cd838fcc2c0bf7c67226425bca9c"}
Sep 17 18:20:13.057090 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:13.057049    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:23.056532 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:23.056498    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:24.175149 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:24.175112    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" containerID="cri-o://7974c185d76434b1c269ffbfefabbcb9b413d9c426c6aaaa9d017c0530e9b9c6" gracePeriod=45
Sep 17 18:20:24.185531 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:24.185496    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main" containerID="cri-o://c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda" gracePeriod=45
Sep 17 18:20:24.340561 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:24.340526    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 17 18:20:24.927383 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:24.927352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4_9d346f08-7fcc-4a63-8db1-5fbf12a26965/modelcar/0.log"
Sep 17 18:20:24.928157 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:24.928133    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerID="7974c185d76434b1c269ffbfefabbcb9b413d9c426c6aaaa9d017c0530e9b9c6" exitCode=0
Sep 17 18:20:24.928276 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:24.928200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerDied","Data":"7974c185d76434b1c269ffbfefabbcb9b413d9c426c6aaaa9d017c0530e9b9c6"}
Sep 17 18:20:25.430800 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.430778    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4_9d346f08-7fcc-4a63-8db1-5fbf12a26965/modelcar/0.log"
Sep 17 18:20:25.433737 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.433718    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:20:25.600040 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600005    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kserve-provision-location\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600040 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tmp-dir\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-model-cache\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600080    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-dshm\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600113    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-home\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600156    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqcrq\" (UniqueName: \"kubernetes.io/projected/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kube-api-access-qqcrq\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tls-certs\") pod \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\" (UID: \"9d346f08-7fcc-4a63-8db1-5fbf12a26965\") "
Sep 17 18:20:25.600546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:25.600546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:25.600824 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.600803    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-home" (OuterVolumeSpecName: "home") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:25.602506 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.602485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:25.602619 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.602511    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kube-api-access-qqcrq" (OuterVolumeSpecName: "kube-api-access-qqcrq") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "kube-api-access-qqcrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:25.602619 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.602605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-dshm" (OuterVolumeSpecName: "dshm") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:25.611890 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.611868    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d346f08-7fcc-4a63-8db1-5fbf12a26965" (UID: "9d346f08-7fcc-4a63-8db1-5fbf12a26965"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:25.701505 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701470    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqcrq\" (UniqueName: \"kubernetes.io/projected/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kube-api-access-qqcrq\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.701505 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701501    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.701505 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701511    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-kserve-provision-location\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.701749 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701519    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.701749 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701529    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.701749 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701536    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.701749 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.701543    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d346f08-7fcc-4a63-8db1-5fbf12a26965-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.934586 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.934501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4_9d346f08-7fcc-4a63-8db1-5fbf12a26965/modelcar/0.log"
Sep 17 18:20:25.935295 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.935269    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"
Sep 17 18:20:25.935421 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.935266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" event={"ID":"9d346f08-7fcc-4a63-8db1-5fbf12a26965","Type":"ContainerDied","Data":"6e366804ce645f84e64363cd0ae343de8f03883c1c0aaf5bd3ea44bc00d82e49"}
Sep 17 18:20:25.935421 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.935418    2561 scope.go:117] "RemoveContainer" containerID="8b53dd4c94ef1746d47b881df830f9292118cd838fcc2c0bf7c67226425bca9c"
Sep 17 18:20:25.944548 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.944526    2561 scope.go:117] "RemoveContainer" containerID="7974c185d76434b1c269ffbfefabbcb9b413d9c426c6aaaa9d017c0530e9b9c6"
Sep 17 18:20:25.952656 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.952619    2561 scope.go:117] "RemoveContainer" containerID="e84785947ea9f70e581cb843e4a7a3166a44ff9f2baa42a66a675c03e8b7b59a"
Sep 17 18:20:25.955842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.955822    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"]
Sep 17 18:20:25.960049 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:25.960027    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4"]
Sep 17 18:20:26.341278 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:26.341239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d28vc4" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:20:27.021377 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:27.021344    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" path="/var/lib/kubelet/pods/9d346f08-7fcc-4a63-8db1-5fbf12a26965/volumes"
Sep 17 18:20:33.056531 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:33.056495    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:33.056926 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:33.056603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:20:43.057418 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:43.057378    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:53.057318 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:53.057239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:54.428980 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.428952    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:20:54.544701 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.544668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvgff\" (UniqueName: \"kubernetes.io/projected/01e44e62-a05e-4f2a-a4a2-a6f639282857-kube-api-access-jvgff\") pod \"01e44e62-a05e-4f2a-a4a2-a6f639282857\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") "
Sep 17 18:20:54.544890 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.544781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e44e62-a05e-4f2a-a4a2-a6f639282857-tls-certs\") pod \"01e44e62-a05e-4f2a-a4a2-a6f639282857\" (UID: \"01e44e62-a05e-4f2a-a4a2-a6f639282857\") "
Sep 17 18:20:54.546899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.546869    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e44e62-a05e-4f2a-a4a2-a6f639282857-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01e44e62-a05e-4f2a-a4a2-a6f639282857" (UID: "01e44e62-a05e-4f2a-a4a2-a6f639282857"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:54.547002 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.546922    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e44e62-a05e-4f2a-a4a2-a6f639282857-kube-api-access-jvgff" (OuterVolumeSpecName: "kube-api-access-jvgff") pod "01e44e62-a05e-4f2a-a4a2-a6f639282857" (UID: "01e44e62-a05e-4f2a-a4a2-a6f639282857"). InnerVolumeSpecName "kube-api-access-jvgff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:54.646117 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.646091    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgff\" (UniqueName: \"kubernetes.io/projected/01e44e62-a05e-4f2a-a4a2-a6f639282857-kube-api-access-jvgff\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:54.646117 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:54.646118    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e44e62-a05e-4f2a-a4a2-a6f639282857-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:20:55.074261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.074231    2561 generic.go:356] "Generic (PLEG): container finished" podID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerID="c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda" exitCode=0
Sep 17 18:20:55.074437 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.074267    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"
Sep 17 18:20:55.074437 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.074298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" event={"ID":"01e44e62-a05e-4f2a-a4a2-a6f639282857","Type":"ContainerDied","Data":"c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda"}
Sep 17 18:20:55.074437 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.074338    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm" event={"ID":"01e44e62-a05e-4f2a-a4a2-a6f639282857","Type":"ContainerDied","Data":"4b14508b87b0f3dabee1e88c6039195f94833851af30f24d846bd125011d2fa0"}
Sep 17 18:20:55.074437 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.074360    2561 scope.go:117] "RemoveContainer" containerID="c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda"
Sep 17 18:20:55.083303 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.083282    2561 scope.go:117] "RemoveContainer" containerID="c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda"
Sep 17 18:20:55.083594 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:20:55.083576    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda\": container with ID starting with c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda not found: ID does not exist" containerID="c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda"
Sep 17 18:20:55.083704 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.083605    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda"} err="failed to get container status \"c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda\": rpc error: code = NotFound desc = could not find container \"c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda\": container with ID starting with c91f28b118b82ef6437985a8a463903b0b596a962987ec452721d089e03ddfda not found: ID does not exist"
Sep 17 18:20:55.091108 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.091083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"]
Sep 17 18:20:55.094040 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:55.094017    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9dkm"]
Sep 17 18:20:57.021533 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:20:57.021496    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" path="/var/lib/kubelet/pods/01e44e62-a05e-4f2a-a4a2-a6f639282857/volumes"
Sep 17 18:21:04.511909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.511876    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"]
Sep 17 18:21:04.513364 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.513093    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="modelcar"
Sep 17 18:21:04.513829 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.513813    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="modelcar"
Sep 17 18:21:04.513959 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.513948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main"
Sep 17 18:21:04.514041 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514033    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main"
Sep 17 18:21:04.514120 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514111    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="modelcar-init"
Sep 17 18:21:04.514187 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514179    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="modelcar-init"
Sep 17 18:21:04.514270 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514262    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerName="main"
Sep 17 18:21:04.514339 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerName="main"
Sep 17 18:21:04.514414 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514405    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main"
Sep 17 18:21:04.514480 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514472    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main"
Sep 17 18:21:04.514748 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514733    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="modelcar"
Sep 17 18:21:04.514855 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514845    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d346f08-7fcc-4a63-8db1-5fbf12a26965" containerName="main"
Sep 17 18:21:04.514938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514919    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b218dc-5da2-4d6b-954f-2e6d95436307" containerName="main"
Sep 17 18:21:04.515067 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.514944    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e44e62-a05e-4f2a-a4a2-a6f639282857" containerName="main"
Sep 17 18:21:04.520070 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.520051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.523818 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.523792    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 18:21:04.524512 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.524481    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 18:21:04.524615 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.524524    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 18:21:04.524615 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.524574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-d8bbg\""
Sep 17 18:21:04.524615 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.524583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"]
Sep 17 18:21:04.524615 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.524605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2wxbb\""
Sep 17 18:21:04.627081 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.627052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560438bb-aa06-4460-b16f-a6878f1b5680-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.627240 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.627086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5p66\" (UniqueName: \"kubernetes.io/projected/560438bb-aa06-4460-b16f-a6878f1b5680-kube-api-access-f5p66\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.728257 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.728219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560438bb-aa06-4460-b16f-a6878f1b5680-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.728257 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.728260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5p66\" (UniqueName: \"kubernetes.io/projected/560438bb-aa06-4460-b16f-a6878f1b5680-kube-api-access-f5p66\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.730738 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.730714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560438bb-aa06-4460-b16f-a6878f1b5680-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.736209 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.736184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5p66\" (UniqueName: \"kubernetes.io/projected/560438bb-aa06-4460-b16f-a6878f1b5680-kube-api-access-f5p66\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:04.832201 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:04.832112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:05.102233 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:05.102160    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"]
Sep 17 18:21:05.104823 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:21:05.104789    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560438bb_aa06_4460_b16f_a6878f1b5680.slice/crio-6e0c25ef01d42c0c4a9918d508f9ba27becc6749926368e8073d36fd0ebe4c5b WatchSource:0}: Error finding container 6e0c25ef01d42c0c4a9918d508f9ba27becc6749926368e8073d36fd0ebe4c5b: Status 404 returned error can't find the container with id 6e0c25ef01d42c0c4a9918d508f9ba27becc6749926368e8073d36fd0ebe4c5b
Sep 17 18:21:05.121266 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:05.121234    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" event={"ID":"560438bb-aa06-4460-b16f-a6878f1b5680","Type":"ContainerStarted","Data":"6e0c25ef01d42c0c4a9918d508f9ba27becc6749926368e8073d36fd0ebe4c5b"}
Sep 17 18:21:06.126912 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:06.126877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" event={"ID":"560438bb-aa06-4460-b16f-a6878f1b5680","Type":"ContainerStarted","Data":"cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b"}
Sep 17 18:21:06.127380 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:06.126937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:21:06.144511 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:06.144465    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" podStartSLOduration=2.144451393 podStartE2EDuration="2.144451393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:06.142969611 +0000 UTC m=+2791.102854900" watchObservedRunningTime="2026-09-17 18:21:06.144451393 +0000 UTC m=+2791.104336662"
Sep 17 18:21:37.132981 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:21:37.132947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:22:35.791504 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.791424    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"]
Sep 17 18:22:35.794880 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.794864    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:35.798735 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.798707    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-qn7pg\""
Sep 17 18:22:35.798735 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.798722    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:22:35.798735 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.798729    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-96cgp\""
Sep 17 18:22:35.798977 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.798707    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:22:35.798977 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.798756    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:22:35.808256 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.808232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"]
Sep 17 18:22:35.890351 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.890311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlg6l\" (UniqueName: \"kubernetes.io/projected/72c557b6-00b3-4266-8803-c50cefc3af73-kube-api-access-xlg6l\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:35.890533 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.890367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c557b6-00b3-4266-8803-c50cefc3af73-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:35.991481 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.991447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlg6l\" (UniqueName: \"kubernetes.io/projected/72c557b6-00b3-4266-8803-c50cefc3af73-kube-api-access-xlg6l\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:35.991664 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.991500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c557b6-00b3-4266-8803-c50cefc3af73-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:35.994030 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.994005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c557b6-00b3-4266-8803-c50cefc3af73-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:35.999450 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:35.999425    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlg6l\" (UniqueName: \"kubernetes.io/projected/72c557b6-00b3-4266-8803-c50cefc3af73-kube-api-access-xlg6l\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:36.105383 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:36.105301    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:36.378282 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:36.378237    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"]
Sep 17 18:22:36.381434 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:22:36.381405    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c557b6_00b3_4266_8803_c50cefc3af73.slice/crio-a766f3c300e09686d7ddbbd5ab1d55c903ba6fc1e9e7b4fd0308c4bc1bddc704 WatchSource:0}: Error finding container a766f3c300e09686d7ddbbd5ab1d55c903ba6fc1e9e7b4fd0308c4bc1bddc704: Status 404 returned error can't find the container with id a766f3c300e09686d7ddbbd5ab1d55c903ba6fc1e9e7b4fd0308c4bc1bddc704
Sep 17 18:22:36.544671 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:36.544614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" event={"ID":"72c557b6-00b3-4266-8803-c50cefc3af73","Type":"ContainerStarted","Data":"e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9"}
Sep 17 18:22:36.544868 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:36.544695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" event={"ID":"72c557b6-00b3-4266-8803-c50cefc3af73","Type":"ContainerStarted","Data":"a766f3c300e09686d7ddbbd5ab1d55c903ba6fc1e9e7b4fd0308c4bc1bddc704"}
Sep 17 18:22:36.544868 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:36.544762    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:22:36.564433 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:22:36.564383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" podStartSLOduration=1.564368613 podStartE2EDuration="1.564368613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:36.5618967 +0000 UTC m=+2881.521781967" watchObservedRunningTime="2026-09-17 18:22:36.564368613 +0000 UTC m=+2881.524253882"
Sep 17 18:23:07.551560 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:07.551529    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:23:09.156555 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:09.156522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"]
Sep 17 18:23:17.133012 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:17.132973    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:24.177948 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:24.177895    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main" containerID="cri-o://cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b" gracePeriod=45
Sep 17 18:23:27.133458 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:27.133418    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:37.133297 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:37.133261    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:37.133715 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:37.133350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:23:47.133246 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:47.133207    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:54.428989 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.428965    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:23:54.551814 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.551781    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5p66\" (UniqueName: \"kubernetes.io/projected/560438bb-aa06-4460-b16f-a6878f1b5680-kube-api-access-f5p66\") pod \"560438bb-aa06-4460-b16f-a6878f1b5680\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") "
Sep 17 18:23:54.551980 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.551891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560438bb-aa06-4460-b16f-a6878f1b5680-tls-certs\") pod \"560438bb-aa06-4460-b16f-a6878f1b5680\" (UID: \"560438bb-aa06-4460-b16f-a6878f1b5680\") "
Sep 17 18:23:54.554144 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.554111    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560438bb-aa06-4460-b16f-a6878f1b5680-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "560438bb-aa06-4460-b16f-a6878f1b5680" (UID: "560438bb-aa06-4460-b16f-a6878f1b5680"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:54.554256 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.554164    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560438bb-aa06-4460-b16f-a6878f1b5680-kube-api-access-f5p66" (OuterVolumeSpecName: "kube-api-access-f5p66") pod "560438bb-aa06-4460-b16f-a6878f1b5680" (UID: "560438bb-aa06-4460-b16f-a6878f1b5680"). InnerVolumeSpecName "kube-api-access-f5p66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:54.652534 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.652506    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560438bb-aa06-4460-b16f-a6878f1b5680-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:23:54.652534 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.652528    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5p66\" (UniqueName: \"kubernetes.io/projected/560438bb-aa06-4460-b16f-a6878f1b5680-kube-api-access-f5p66\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:23:54.896734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.896613    2561 generic.go:356] "Generic (PLEG): container finished" podID="560438bb-aa06-4460-b16f-a6878f1b5680" containerID="cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b" exitCode=0
Sep 17 18:23:54.896734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.896687    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"
Sep 17 18:23:54.896734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.896706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" event={"ID":"560438bb-aa06-4460-b16f-a6878f1b5680","Type":"ContainerDied","Data":"cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b"}
Sep 17 18:23:54.897019 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.896774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d" event={"ID":"560438bb-aa06-4460-b16f-a6878f1b5680","Type":"ContainerDied","Data":"6e0c25ef01d42c0c4a9918d508f9ba27becc6749926368e8073d36fd0ebe4c5b"}
Sep 17 18:23:54.897019 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.896803    2561 scope.go:117] "RemoveContainer" containerID="cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b"
Sep 17 18:23:54.905748 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.905732    2561 scope.go:117] "RemoveContainer" containerID="cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b"
Sep 17 18:23:54.905995 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:23:54.905978    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b\": container with ID starting with cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b not found: ID does not exist" containerID="cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b"
Sep 17 18:23:54.906045 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.906004    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b"} err="failed to get container status \"cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b\": rpc error: code = NotFound desc = could not find container \"cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b\": container with ID starting with cc066e8c2f935b68bda096cde5dff89cafcc1d8b48205adbce21433b4e5ee12b not found: ID does not exist"
Sep 17 18:23:54.918786 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.918763    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"]
Sep 17 18:23:54.922339 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:54.922318    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bhp5d"]
Sep 17 18:23:55.019948 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:55.019923    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" path="/var/lib/kubelet/pods/560438bb-aa06-4460-b16f-a6878f1b5680/volumes"
Sep 17 18:23:57.663951 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:23:57.663921    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"]
Sep 17 18:24:07.550872 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:07.550833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:12.683377 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:12.683305    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main" containerID="cri-o://e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9" gracePeriod=45
Sep 17 18:24:17.550953 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:17.550908    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:19.707008 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.706966    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"]
Sep 17 18:24:19.707521 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.707501    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main"
Sep 17 18:24:19.707598 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.707525    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main"
Sep 17 18:24:19.707677 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.707614    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="560438bb-aa06-4460-b16f-a6878f1b5680" containerName="main"
Sep 17 18:24:19.710508 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.710486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:19.714297 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.714276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:24:19.714420 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.714310    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 18:24:19.714420 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.714375    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-l9zxl\""
Sep 17 18:24:19.714548 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.714492    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:24:19.714671 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.714626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-9pbts\""
Sep 17 18:24:19.720432 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.720411    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"]
Sep 17 18:24:19.770852 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.770825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprws\" (UniqueName: \"kubernetes.io/projected/7b601231-da43-4de4-b5df-0ab18e8cbd5a-kube-api-access-sprws\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:19.770992 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.770887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b601231-da43-4de4-b5df-0ab18e8cbd5a-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:19.871874 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.871841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b601231-da43-4de4-b5df-0ab18e8cbd5a-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:19.872042 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.871936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprws\" (UniqueName: \"kubernetes.io/projected/7b601231-da43-4de4-b5df-0ab18e8cbd5a-kube-api-access-sprws\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:19.874444 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.874418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b601231-da43-4de4-b5df-0ab18e8cbd5a-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:19.881090 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:19.881064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprws\" (UniqueName: \"kubernetes.io/projected/7b601231-da43-4de4-b5df-0ab18e8cbd5a-kube-api-access-sprws\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:20.020922 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:20.020886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:20.292969 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:20.292891    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"]
Sep 17 18:24:20.296466 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:24:20.296429    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b601231_da43_4de4_b5df_0ab18e8cbd5a.slice/crio-e0e913226198d618fcebbd865460da298a993f0d0865b6bc3b4123377e480bd9 WatchSource:0}: Error finding container e0e913226198d618fcebbd865460da298a993f0d0865b6bc3b4123377e480bd9: Status 404 returned error can't find the container with id e0e913226198d618fcebbd865460da298a993f0d0865b6bc3b4123377e480bd9
Sep 17 18:24:21.013867 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:21.013826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" event={"ID":"7b601231-da43-4de4-b5df-0ab18e8cbd5a","Type":"ContainerStarted","Data":"02f71b9d6238fdef7b59389d9cf1ea70de44743d99c26c84dda29ffeb925825c"}
Sep 17 18:24:21.014261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:21.013893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" event={"ID":"7b601231-da43-4de4-b5df-0ab18e8cbd5a","Type":"ContainerStarted","Data":"e0e913226198d618fcebbd865460da298a993f0d0865b6bc3b4123377e480bd9"}
Sep 17 18:24:21.014261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:21.013934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:21.033405 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:21.033358    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" podStartSLOduration=2.033344691 podStartE2EDuration="2.033344691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:21.031077139 +0000 UTC m=+2985.990962411" watchObservedRunningTime="2026-09-17 18:24:21.033344691 +0000 UTC m=+2985.993229958"
Sep 17 18:24:27.550806 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:27.550766    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:27.551256 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:27.550884    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:24:37.178117 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:37.178089    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:24:37.184691 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:37.184671    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:24:37.551295 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:37.551262    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:42.921591 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:42.921567    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:24:43.065888 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.065848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c557b6-00b3-4266-8803-c50cefc3af73-tls-certs\") pod \"72c557b6-00b3-4266-8803-c50cefc3af73\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") "
Sep 17 18:24:43.066080 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.065910    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlg6l\" (UniqueName: \"kubernetes.io/projected/72c557b6-00b3-4266-8803-c50cefc3af73-kube-api-access-xlg6l\") pod \"72c557b6-00b3-4266-8803-c50cefc3af73\" (UID: \"72c557b6-00b3-4266-8803-c50cefc3af73\") "
Sep 17 18:24:43.068223 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.068196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c557b6-00b3-4266-8803-c50cefc3af73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72c557b6-00b3-4266-8803-c50cefc3af73" (UID: "72c557b6-00b3-4266-8803-c50cefc3af73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:43.068367 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.068355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c557b6-00b3-4266-8803-c50cefc3af73-kube-api-access-xlg6l" (OuterVolumeSpecName: "kube-api-access-xlg6l") pod "72c557b6-00b3-4266-8803-c50cefc3af73" (UID: "72c557b6-00b3-4266-8803-c50cefc3af73"). InnerVolumeSpecName "kube-api-access-xlg6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:43.110555 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.110523    2561 generic.go:356] "Generic (PLEG): container finished" podID="72c557b6-00b3-4266-8803-c50cefc3af73" containerID="e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9" exitCode=0
Sep 17 18:24:43.110720 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.110564    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"
Sep 17 18:24:43.110720 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.110603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" event={"ID":"72c557b6-00b3-4266-8803-c50cefc3af73","Type":"ContainerDied","Data":"e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9"}
Sep 17 18:24:43.110720 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.110676    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff" event={"ID":"72c557b6-00b3-4266-8803-c50cefc3af73","Type":"ContainerDied","Data":"a766f3c300e09686d7ddbbd5ab1d55c903ba6fc1e9e7b4fd0308c4bc1bddc704"}
Sep 17 18:24:43.110720 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.110700    2561 scope.go:117] "RemoveContainer" containerID="e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9"
Sep 17 18:24:43.119542 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.119517    2561 scope.go:117] "RemoveContainer" containerID="e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9"
Sep 17 18:24:43.119835 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:24:43.119819    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9\": container with ID starting with e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9 not found: ID does not exist" containerID="e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9"
Sep 17 18:24:43.119888 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.119843    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9"} err="failed to get container status \"e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9\": rpc error: code = NotFound desc = could not find container \"e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9\": container with ID starting with e5a6ec41a6ba2239c3807c5c1997af6a44977da717554eccfdcd92f1a8b26ad9 not found: ID does not exist"
Sep 17 18:24:43.132669 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.132644    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"]
Sep 17 18:24:43.136484 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.136465    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche7vbff"]
Sep 17 18:24:43.166774 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.166753    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c557b6-00b3-4266-8803-c50cefc3af73-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:24:43.166774 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:43.166773    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlg6l\" (UniqueName: \"kubernetes.io/projected/72c557b6-00b3-4266-8803-c50cefc3af73-kube-api-access-xlg6l\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:24:45.019977 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:45.019939    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" path="/var/lib/kubelet/pods/72c557b6-00b3-4266-8803-c50cefc3af73/volumes"
Sep 17 18:24:51.816383 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.816353    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"]
Sep 17 18:24:51.816819 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.816800    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main"
Sep 17 18:24:51.816878 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.816824    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main"
Sep 17 18:24:51.816943 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.816931    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c557b6-00b3-4266-8803-c50cefc3af73" containerName="main"
Sep 17 18:24:51.821179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.821162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:51.828402 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.828361    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jhjnz\""
Sep 17 18:24:51.828402 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.828361    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:24:51.828648 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.828416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-jhg8m\""
Sep 17 18:24:51.828648 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.828391    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:24:51.828648 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.828363    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:24:51.832708 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.832685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613b1700-c6e1-4716-8b4b-41b429c538a1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:51.832791 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.832778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwx5\" (UniqueName: \"kubernetes.io/projected/613b1700-c6e1-4716-8b4b-41b429c538a1-kube-api-access-7pwx5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:51.833212 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.833195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"]
Sep 17 18:24:51.933943 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.933911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613b1700-c6e1-4716-8b4b-41b429c538a1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:51.934075 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.933970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwx5\" (UniqueName: \"kubernetes.io/projected/613b1700-c6e1-4716-8b4b-41b429c538a1-kube-api-access-7pwx5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:51.936446 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.936419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613b1700-c6e1-4716-8b4b-41b429c538a1-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:51.942197 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:51.942174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwx5\" (UniqueName: \"kubernetes.io/projected/613b1700-c6e1-4716-8b4b-41b429c538a1-kube-api-access-7pwx5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:52.020149 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:52.020127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:24:52.131058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:52.130970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:52.436696 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:52.436550    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"]
Sep 17 18:24:53.164352 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:53.164309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" event={"ID":"613b1700-c6e1-4716-8b4b-41b429c538a1","Type":"ContainerStarted","Data":"bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d"}
Sep 17 18:24:53.164872 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:53.164382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" event={"ID":"613b1700-c6e1-4716-8b4b-41b429c538a1","Type":"ContainerStarted","Data":"2f71f1add1ccb7801ab6241902ee2d82851888f6c26b622a56789013d50ed066"}
Sep 17 18:24:53.164872 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:53.164577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:24:53.212725 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:24:53.212679    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podStartSLOduration=2.212662627 podStartE2EDuration="2.212662627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:53.211090893 +0000 UTC m=+3018.170976166" watchObservedRunningTime="2026-09-17 18:24:53.212662627 +0000 UTC m=+3018.172547897"
Sep 17 18:25:24.171843 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:25:24.171768    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:26:24.570233 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:24.570202    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"]
Sep 17 18:26:32.019377 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:32.019339    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:39.590449 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:39.590391    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main" containerID="cri-o://02f71b9d6238fdef7b59389d9cf1ea70de44743d99c26c84dda29ffeb925825c" gracePeriod=45
Sep 17 18:26:42.019842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:42.019798    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:52.020183 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:52.020107    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:52.020523 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:52.020213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:26:53.615185 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:53.615149    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"]
Sep 17 18:26:54.171893 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:26:54.171840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:27:02.020129 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:02.020090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:27:04.172053 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:04.172018    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:27:08.636684 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:08.636625    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main" containerID="cri-o://bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d" gracePeriod=45
Sep 17 18:27:09.782376 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.782340    2561 generic.go:356] "Generic (PLEG): container finished" podID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerID="02f71b9d6238fdef7b59389d9cf1ea70de44743d99c26c84dda29ffeb925825c" exitCode=0
Sep 17 18:27:09.782770 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.782420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" event={"ID":"7b601231-da43-4de4-b5df-0ab18e8cbd5a","Type":"ContainerDied","Data":"02f71b9d6238fdef7b59389d9cf1ea70de44743d99c26c84dda29ffeb925825c"}
Sep 17 18:27:09.840789 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.840764    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:27:09.890913 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.890846    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sprws\" (UniqueName: \"kubernetes.io/projected/7b601231-da43-4de4-b5df-0ab18e8cbd5a-kube-api-access-sprws\") pod \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") "
Sep 17 18:27:09.890913 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.890875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b601231-da43-4de4-b5df-0ab18e8cbd5a-tls-certs\") pod \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\" (UID: \"7b601231-da43-4de4-b5df-0ab18e8cbd5a\") "
Sep 17 18:27:09.892971 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.892940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b601231-da43-4de4-b5df-0ab18e8cbd5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b601231-da43-4de4-b5df-0ab18e8cbd5a" (UID: "7b601231-da43-4de4-b5df-0ab18e8cbd5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:09.893079 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.892970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b601231-da43-4de4-b5df-0ab18e8cbd5a-kube-api-access-sprws" (OuterVolumeSpecName: "kube-api-access-sprws") pod "7b601231-da43-4de4-b5df-0ab18e8cbd5a" (UID: "7b601231-da43-4de4-b5df-0ab18e8cbd5a"). InnerVolumeSpecName "kube-api-access-sprws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:09.992352 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.992320    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprws\" (UniqueName: \"kubernetes.io/projected/7b601231-da43-4de4-b5df-0ab18e8cbd5a-kube-api-access-sprws\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:27:09.992352 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:09.992346    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b601231-da43-4de4-b5df-0ab18e8cbd5a-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:27:10.788129 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:10.788101    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"
Sep 17 18:27:10.788598 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:10.788106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7" event={"ID":"7b601231-da43-4de4-b5df-0ab18e8cbd5a","Type":"ContainerDied","Data":"e0e913226198d618fcebbd865460da298a993f0d0865b6bc3b4123377e480bd9"}
Sep 17 18:27:10.788598 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:10.788235    2561 scope.go:117] "RemoveContainer" containerID="02f71b9d6238fdef7b59389d9cf1ea70de44743d99c26c84dda29ffeb925825c"
Sep 17 18:27:10.807703 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:10.807679    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"]
Sep 17 18:27:10.812048 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:10.812027    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-c8nn7"]
Sep 17 18:27:11.021063 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:11.021034    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" path="/var/lib/kubelet/pods/7b601231-da43-4de4-b5df-0ab18e8cbd5a/volumes"
Sep 17 18:27:14.172473 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:14.172437    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:27:14.172874 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:14.172535    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:27:24.171572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:24.171538    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:27:34.172237 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:34.172198    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:27:35.067274 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.067227    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"]
Sep 17 18:27:35.067583 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.067571    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main"
Sep 17 18:27:35.067656 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.067584    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main"
Sep 17 18:27:35.067656 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.067653    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b601231-da43-4de4-b5df-0ab18e8cbd5a" containerName="main"
Sep 17 18:27:35.071857 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.071835    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.075436 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.075394    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-v6v88\""
Sep 17 18:27:35.075585 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.075394    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:27:35.075585 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.075453    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:27:35.075585 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.075454    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:27:35.075585 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.075459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-fjttw\""
Sep 17 18:27:35.080936 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.080915    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"]
Sep 17 18:27:35.108613 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.108590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e54b9-dcdf-4777-aa95-90b1fa809db7-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.108843 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.108826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4n5\" (UniqueName: \"kubernetes.io/projected/f39e54b9-dcdf-4777-aa95-90b1fa809db7-kube-api-access-9k4n5\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.209836 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.209790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4n5\" (UniqueName: \"kubernetes.io/projected/f39e54b9-dcdf-4777-aa95-90b1fa809db7-kube-api-access-9k4n5\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.210315 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.210126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e54b9-dcdf-4777-aa95-90b1fa809db7-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.213032 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.213003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e54b9-dcdf-4777-aa95-90b1fa809db7-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.217670 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.217647    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4n5\" (UniqueName: \"kubernetes.io/projected/f39e54b9-dcdf-4777-aa95-90b1fa809db7-kube-api-access-9k4n5\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.382104 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.382020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.655040 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.655007    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"]
Sep 17 18:27:35.658388 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:27:35.658348    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39e54b9_dcdf_4777_aa95_90b1fa809db7.slice/crio-279326eaf79ac3fe49549d9c0d7b8215554e80f9844b8332c6018a4a7467d662 WatchSource:0}: Error finding container 279326eaf79ac3fe49549d9c0d7b8215554e80f9844b8332c6018a4a7467d662: Status 404 returned error can't find the container with id 279326eaf79ac3fe49549d9c0d7b8215554e80f9844b8332c6018a4a7467d662
Sep 17 18:27:35.899937 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.899901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" event={"ID":"f39e54b9-dcdf-4777-aa95-90b1fa809db7","Type":"ContainerStarted","Data":"897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471"}
Sep 17 18:27:35.900123 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.899959    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" event={"ID":"f39e54b9-dcdf-4777-aa95-90b1fa809db7","Type":"ContainerStarted","Data":"279326eaf79ac3fe49549d9c0d7b8215554e80f9844b8332c6018a4a7467d662"}
Sep 17 18:27:35.900123 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.899984    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:27:35.920149 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:35.920060    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podStartSLOduration=0.920044647 podStartE2EDuration="920.044647ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:35.918182673 +0000 UTC m=+3180.878067942" watchObservedRunningTime="2026-09-17 18:27:35.920044647 +0000 UTC m=+3180.879929915"
Sep 17 18:27:38.878260 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.878237    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:27:38.914256 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.914194    2561 generic.go:356] "Generic (PLEG): container finished" podID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerID="bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d" exitCode=0
Sep 17 18:27:38.914256 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.914229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" event={"ID":"613b1700-c6e1-4716-8b4b-41b429c538a1","Type":"ContainerDied","Data":"bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d"}
Sep 17 18:27:38.914404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.914267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq" event={"ID":"613b1700-c6e1-4716-8b4b-41b429c538a1","Type":"ContainerDied","Data":"2f71f1add1ccb7801ab6241902ee2d82851888f6c26b622a56789013d50ed066"}
Sep 17 18:27:38.914404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.914292    2561 scope.go:117] "RemoveContainer" containerID="bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d"
Sep 17 18:27:38.914404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.914230    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"
Sep 17 18:27:38.922143 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.922125    2561 scope.go:117] "RemoveContainer" containerID="bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d"
Sep 17 18:27:38.922403 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:27:38.922375    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d\": container with ID starting with bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d not found: ID does not exist" containerID="bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d"
Sep 17 18:27:38.922490 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.922400    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d"} err="failed to get container status \"bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d\": rpc error: code = NotFound desc = could not find container \"bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d\": container with ID starting with bc939c421dcb1469c6fd4f7afbb867572376598b5f9e529082245e03d961285d not found: ID does not exist"
Sep 17 18:27:38.936904 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.936887    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pwx5\" (UniqueName: \"kubernetes.io/projected/613b1700-c6e1-4716-8b4b-41b429c538a1-kube-api-access-7pwx5\") pod \"613b1700-c6e1-4716-8b4b-41b429c538a1\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") "
Sep 17 18:27:38.936982 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.936951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613b1700-c6e1-4716-8b4b-41b429c538a1-tls-certs\") pod \"613b1700-c6e1-4716-8b4b-41b429c538a1\" (UID: \"613b1700-c6e1-4716-8b4b-41b429c538a1\") "
Sep 17 18:27:38.938926 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.938900    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613b1700-c6e1-4716-8b4b-41b429c538a1-kube-api-access-7pwx5" (OuterVolumeSpecName: "kube-api-access-7pwx5") pod "613b1700-c6e1-4716-8b4b-41b429c538a1" (UID: "613b1700-c6e1-4716-8b4b-41b429c538a1"). InnerVolumeSpecName "kube-api-access-7pwx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:38.939016 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:38.938997    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613b1700-c6e1-4716-8b4b-41b429c538a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "613b1700-c6e1-4716-8b4b-41b429c538a1" (UID: "613b1700-c6e1-4716-8b4b-41b429c538a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:39.037952 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:39.037915    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwx5\" (UniqueName: \"kubernetes.io/projected/613b1700-c6e1-4716-8b4b-41b429c538a1-kube-api-access-7pwx5\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:27:39.037952 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:39.037947    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613b1700-c6e1-4716-8b4b-41b429c538a1-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:27:39.235236 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:39.234184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"]
Sep 17 18:27:39.238835 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:39.238804    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-z2gdq"]
Sep 17 18:27:41.020853 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:27:41.020822    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" path="/var/lib/kubelet/pods/613b1700-c6e1-4716-8b4b-41b429c538a1/volumes"
Sep 17 18:28:06.906894 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:28:06.906865    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:29:37.206087 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:29:37.206061    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:29:37.212690 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:29:37.212670    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:29:53.863889 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:29:53.863799    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"]
Sep 17 18:29:56.906387 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:29:56.906345    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:06.906649 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:06.906592    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:08.888521 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:08.888480    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main" containerID="cri-o://897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471" gracePeriod=45
Sep 17 18:30:16.906722 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:16.906682    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:16.907202 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:16.906797    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:30:26.906168 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:26.906130    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:36.905961 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:36.905923    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:39.132262 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.132238    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:30:39.286512 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.286478    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k4n5\" (UniqueName: \"kubernetes.io/projected/f39e54b9-dcdf-4777-aa95-90b1fa809db7-kube-api-access-9k4n5\") pod \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") "
Sep 17 18:30:39.286734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.286539    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e54b9-dcdf-4777-aa95-90b1fa809db7-tls-certs\") pod \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\" (UID: \"f39e54b9-dcdf-4777-aa95-90b1fa809db7\") "
Sep 17 18:30:39.288803 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.288775    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39e54b9-dcdf-4777-aa95-90b1fa809db7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f39e54b9-dcdf-4777-aa95-90b1fa809db7" (UID: "f39e54b9-dcdf-4777-aa95-90b1fa809db7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:39.288935 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.288801    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39e54b9-dcdf-4777-aa95-90b1fa809db7-kube-api-access-9k4n5" (OuterVolumeSpecName: "kube-api-access-9k4n5") pod "f39e54b9-dcdf-4777-aa95-90b1fa809db7" (UID: "f39e54b9-dcdf-4777-aa95-90b1fa809db7"). InnerVolumeSpecName "kube-api-access-9k4n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:39.387257 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.387220    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4n5\" (UniqueName: \"kubernetes.io/projected/f39e54b9-dcdf-4777-aa95-90b1fa809db7-kube-api-access-9k4n5\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.387257 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.387252    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f39e54b9-dcdf-4777-aa95-90b1fa809db7-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.723734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.723626    2561 generic.go:356] "Generic (PLEG): container finished" podID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerID="897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471" exitCode=0
Sep 17 18:30:39.723734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.723692    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"
Sep 17 18:30:39.723734 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.723702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" event={"ID":"f39e54b9-dcdf-4777-aa95-90b1fa809db7","Type":"ContainerDied","Data":"897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471"}
Sep 17 18:30:39.723958 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.723755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm" event={"ID":"f39e54b9-dcdf-4777-aa95-90b1fa809db7","Type":"ContainerDied","Data":"279326eaf79ac3fe49549d9c0d7b8215554e80f9844b8332c6018a4a7467d662"}
Sep 17 18:30:39.723958 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.723784    2561 scope.go:117] "RemoveContainer" containerID="897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471"
Sep 17 18:30:39.733569 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.733546    2561 scope.go:117] "RemoveContainer" containerID="897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471"
Sep 17 18:30:39.733882 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:30:39.733858    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471\": container with ID starting with 897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471 not found: ID does not exist" containerID="897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471"
Sep 17 18:30:39.733983 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.733888    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471"} err="failed to get container status \"897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471\": rpc error: code = NotFound desc = could not find container \"897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471\": container with ID starting with 897de0c111b89dbaad0264c9913c8fcffa99f3a42fd0dac4f338b14d16e02471 not found: ID does not exist"
Sep 17 18:30:39.747404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.747382    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"]
Sep 17 18:30:39.753072 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:39.753043    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche6p2pm"]
Sep 17 18:30:41.019958 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:30:41.019926    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" path="/var/lib/kubelet/pods/f39e54b9-dcdf-4777-aa95-90b1fa809db7/volumes"
Sep 17 18:31:11.577400 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577367    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"]
Sep 17 18:31:11.577938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577706    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main"
Sep 17 18:31:11.577938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577719    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main"
Sep 17 18:31:11.577938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577730    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main"
Sep 17 18:31:11.577938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577735    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main"
Sep 17 18:31:11.577938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577783    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="613b1700-c6e1-4716-8b4b-41b429c538a1" containerName="main"
Sep 17 18:31:11.577938 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.577793    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39e54b9-dcdf-4777-aa95-90b1fa809db7" containerName="main"
Sep 17 18:31:11.580692 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.580673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.584392 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.584359    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:31:11.584570 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.584549    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-gcrsm\""
Sep 17 18:31:11.584670 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.584619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:31:11.584746 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.584673    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:31:11.584746 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.584695    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-76g2p\""
Sep 17 18:31:11.591866 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.591844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"]
Sep 17 18:31:11.660229 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.660194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4ea634-8c58-443b-984b-5fe331d95b94-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.660392 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.660233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blf57\" (UniqueName: \"kubernetes.io/projected/9f4ea634-8c58-443b-984b-5fe331d95b94-kube-api-access-blf57\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.761095 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.761068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4ea634-8c58-443b-984b-5fe331d95b94-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.761260 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.761105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blf57\" (UniqueName: \"kubernetes.io/projected/9f4ea634-8c58-443b-984b-5fe331d95b94-kube-api-access-blf57\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.763576 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.763558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4ea634-8c58-443b-984b-5fe331d95b94-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.768759 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.768739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blf57\" (UniqueName: \"kubernetes.io/projected/9f4ea634-8c58-443b-984b-5fe331d95b94-kube-api-access-blf57\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:11.892770 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:11.892687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:12.161514 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:12.161486    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"]
Sep 17 18:31:12.164184 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:31:12.164158    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4ea634_8c58_443b_984b_5fe331d95b94.slice/crio-24f984ab453ac6ff2cbefc9cd9719a314874d69ec112f56424f0119758f4e289 WatchSource:0}: Error finding container 24f984ab453ac6ff2cbefc9cd9719a314874d69ec112f56424f0119758f4e289: Status 404 returned error can't find the container with id 24f984ab453ac6ff2cbefc9cd9719a314874d69ec112f56424f0119758f4e289
Sep 17 18:31:12.880388 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:12.880354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" event={"ID":"9f4ea634-8c58-443b-984b-5fe331d95b94","Type":"ContainerStarted","Data":"dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f"}
Sep 17 18:31:12.880799 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:12.880405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" event={"ID":"9f4ea634-8c58-443b-984b-5fe331d95b94","Type":"ContainerStarted","Data":"24f984ab453ac6ff2cbefc9cd9719a314874d69ec112f56424f0119758f4e289"}
Sep 17 18:31:12.880799 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:12.880436    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:31:12.913239 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:12.913190    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podStartSLOduration=1.9131751289999999 podStartE2EDuration="1.913175129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:12.911571407 +0000 UTC m=+3397.871456677" watchObservedRunningTime="2026-09-17 18:31:12.913175129 +0000 UTC m=+3397.873060398"
Sep 17 18:31:43.886446 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:31:43.886413    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:33:22.010361 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:22.010324    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"]
Sep 17 18:33:23.885818 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:23.885785    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:33:33.886869 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:33.886832    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:33:37.029339 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:37.029296    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main" containerID="cri-o://dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f" gracePeriod=45
Sep 17 18:33:43.885788 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:43.885743    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:33:43.886327 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:43.885850    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:33:53.886065 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:33:53.886030    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:03.886420 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:03.886374    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:07.278131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.278103    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:34:07.444878 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.444777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4ea634-8c58-443b-984b-5fe331d95b94-tls-certs\") pod \"9f4ea634-8c58-443b-984b-5fe331d95b94\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") "
Sep 17 18:34:07.444878 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.444843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blf57\" (UniqueName: \"kubernetes.io/projected/9f4ea634-8c58-443b-984b-5fe331d95b94-kube-api-access-blf57\") pod \"9f4ea634-8c58-443b-984b-5fe331d95b94\" (UID: \"9f4ea634-8c58-443b-984b-5fe331d95b94\") "
Sep 17 18:34:07.447034 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.446999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4ea634-8c58-443b-984b-5fe331d95b94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f4ea634-8c58-443b-984b-5fe331d95b94" (UID: "9f4ea634-8c58-443b-984b-5fe331d95b94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:34:07.447147 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.447069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4ea634-8c58-443b-984b-5fe331d95b94-kube-api-access-blf57" (OuterVolumeSpecName: "kube-api-access-blf57") pod "9f4ea634-8c58-443b-984b-5fe331d95b94" (UID: "9f4ea634-8c58-443b-984b-5fe331d95b94"). InnerVolumeSpecName "kube-api-access-blf57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:07.545978 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.545942    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4ea634-8c58-443b-984b-5fe331d95b94-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:34:07.545978 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.545974    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blf57\" (UniqueName: \"kubernetes.io/projected/9f4ea634-8c58-443b-984b-5fe331d95b94-kube-api-access-blf57\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:34:07.664648 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.664600    2561 generic.go:356] "Generic (PLEG): container finished" podID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerID="dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f" exitCode=0
Sep 17 18:34:07.664848 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.664676    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"
Sep 17 18:34:07.664848 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.664687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" event={"ID":"9f4ea634-8c58-443b-984b-5fe331d95b94","Type":"ContainerDied","Data":"dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f"}
Sep 17 18:34:07.664848 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.664741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w" event={"ID":"9f4ea634-8c58-443b-984b-5fe331d95b94","Type":"ContainerDied","Data":"24f984ab453ac6ff2cbefc9cd9719a314874d69ec112f56424f0119758f4e289"}
Sep 17 18:34:07.664848 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.664764    2561 scope.go:117] "RemoveContainer" containerID="dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f"
Sep 17 18:34:07.673547 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.673531    2561 scope.go:117] "RemoveContainer" containerID="dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f"
Sep 17 18:34:07.673826 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:34:07.673803    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f\": container with ID starting with dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f not found: ID does not exist" containerID="dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f"
Sep 17 18:34:07.673909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.673833    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f"} err="failed to get container status \"dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f\": rpc error: code = NotFound desc = could not find container \"dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f\": container with ID starting with dfe3ab0c0518353a8401dbdc6ff353c2412baa610351d8459499cb6eab72c27f not found: ID does not exist"
Sep 17 18:34:07.697122 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.697028    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"]
Sep 17 18:34:07.698557 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:07.698533    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48fjg2w"]
Sep 17 18:34:09.020190 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:09.020159    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" path="/var/lib/kubelet/pods/9f4ea634-8c58-443b-984b-5fe331d95b94/volumes"
Sep 17 18:34:12.482678 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.482643    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"]
Sep 17 18:34:12.483148 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.483132    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main"
Sep 17 18:34:12.483218 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.483151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main"
Sep 17 18:34:12.483281 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.483268    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f4ea634-8c58-443b-984b-5fe331d95b94" containerName="main"
Sep 17 18:34:12.486618 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.486589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.489431 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.489407    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:34:12.489594 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.489458    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:34:12.489594 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.489579    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:34:12.490516 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.490498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-fwrdg\""
Sep 17 18:34:12.500706 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.500685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"]
Sep 17 18:34:12.589409 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589409 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589682 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589682 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589682 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblld\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-kube-api-access-bblld\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589682 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589851 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589851 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589753    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.589851 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.589769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691098 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691098 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblld\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-kube-api-access-bblld\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.691995 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.692105 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.692034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.692159 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.691991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.692318 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.692297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.693044 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.693016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.698650 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.694505    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.698650 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.694685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.703407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.703386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.703407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.703396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblld\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-kube-api-access-bblld\") pod \"router-gateway-2-openshift-default-844b774c6-hvgm6\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:12.797568 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:12.797500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:13.096948 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.096919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"]
Sep 17 18:34:13.099427 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:34:13.099399    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcefdce_b19a_40df_bcab_3963b1b1c2b8.slice/crio-f86bfa64d6cfedc6eee5bc1957f7a83baafc12b31e00dce361c701966fd2c25e WatchSource:0}: Error finding container f86bfa64d6cfedc6eee5bc1957f7a83baafc12b31e00dce361c701966fd2c25e: Status 404 returned error can't find the container with id f86bfa64d6cfedc6eee5bc1957f7a83baafc12b31e00dce361c701966fd2c25e
Sep 17 18:34:13.101419 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.101388    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:34:13.101516 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.101450    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:34:13.101516 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.101479    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:34:13.695647 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.695606    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" event={"ID":"4bcefdce-b19a-40df-bcab-3963b1b1c2b8","Type":"ContainerStarted","Data":"cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"}
Sep 17 18:34:13.696032 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.695679    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" event={"ID":"4bcefdce-b19a-40df-bcab-3963b1b1c2b8","Type":"ContainerStarted","Data":"f86bfa64d6cfedc6eee5bc1957f7a83baafc12b31e00dce361c701966fd2c25e"}
Sep 17 18:34:13.722274 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.722218    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podStartSLOduration=1.722200511 podStartE2EDuration="1.722200511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:13.720898963 +0000 UTC m=+3578.680784232" watchObservedRunningTime="2026-09-17 18:34:13.722200511 +0000 UTC m=+3578.682085782"
Sep 17 18:34:13.798015 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:13.797982    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:14.802739 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:14.802710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:15.705583 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:15.705549    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:15.706446 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:15.706423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:24.748452 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:24.748421    2561 generic.go:356] "Generic (PLEG): container finished" podID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerID="cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2" exitCode=0
Sep 17 18:34:24.748995 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:24.748493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" event={"ID":"4bcefdce-b19a-40df-bcab-3963b1b1c2b8","Type":"ContainerDied","Data":"cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"}
Sep 17 18:34:24.748995 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:24.748983    2561 scope.go:117] "RemoveContainer" containerID="cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"
Sep 17 18:34:24.749949 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:24.749922    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:34:24.750039 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:24.749987    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:34:24.750039 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:24.750021    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:34:25.757258 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:25.757218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" event={"ID":"4bcefdce-b19a-40df-bcab-3963b1b1c2b8","Type":"ContainerStarted","Data":"f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68"}
Sep 17 18:34:25.798526 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:25.798493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:25.798731 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:25.798693    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 17 18:34:26.797903 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:26.797860    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 17 18:34:27.798170 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:27.798124    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 17 18:34:28.798418 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:28.798377    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 17 18:34:29.798089 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:29.798045    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 17 18:34:30.706127 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:30.706083    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:30.798789 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:30.798745    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 17 18:34:31.801763 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:31.801733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:31.802786 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:31.802764    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:34:37.234156 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:37.234125    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:34:37.242368 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:37.242344    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:34:38.594425 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.594392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"]
Sep 17 18:34:38.599030 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.599011    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.602724 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.602700    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-bwhqj\""
Sep 17 18:34:38.602850 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.602700    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:34:38.602850 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.602724    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-jstcm\""
Sep 17 18:34:38.608261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.608239    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"]
Sep 17 18:34:38.748558 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.748525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdj7\" (UniqueName: \"kubernetes.io/projected/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-kube-api-access-twdj7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.748704 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.748566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.849615 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.849524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdj7\" (UniqueName: \"kubernetes.io/projected/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-kube-api-access-twdj7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.849615 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.849572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.852263 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.852227    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.857881 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.857847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdj7\" (UniqueName: \"kubernetes.io/projected/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-kube-api-access-twdj7\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:38.910148 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:38.910114    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:39.209474 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:39.209449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"]
Sep 17 18:34:39.211468 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:34:39.211443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d71a0d_bcaa_4ed2_a09e_2d11cad877e8.slice/crio-3ab0a1f6a48884079c438ece7b3fd34636999c08ac643205053bd7ef0e41adc7 WatchSource:0}: Error finding container 3ab0a1f6a48884079c438ece7b3fd34636999c08ac643205053bd7ef0e41adc7: Status 404 returned error can't find the container with id 3ab0a1f6a48884079c438ece7b3fd34636999c08ac643205053bd7ef0e41adc7
Sep 17 18:34:39.827853 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:39.827821    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" event={"ID":"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8","Type":"ContainerStarted","Data":"4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b"}
Sep 17 18:34:39.828230 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:39.827874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" event={"ID":"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8","Type":"ContainerStarted","Data":"3ab0a1f6a48884079c438ece7b3fd34636999c08ac643205053bd7ef0e41adc7"}
Sep 17 18:34:39.828230 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:39.827905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:34:39.855911 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:34:39.855864    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podStartSLOduration=1.8558437859999999 podStartE2EDuration="1.855843786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:39.854784378 +0000 UTC m=+3604.814669648" watchObservedRunningTime="2026-09-17 18:34:39.855843786 +0000 UTC m=+3604.815729057"
Sep 17 18:35:10.835180 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:35:10.835149    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:36:57.226969 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:36:57.226924    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"]
Sep 17 18:37:00.834790 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:00.834748    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:10.835243 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:10.835200    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:12.246976 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:12.246923    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main" containerID="cri-o://4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b" gracePeriod=45
Sep 17 18:37:20.835326 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:20.835239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:20.835915 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:20.835337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:37:30.834689 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:30.834627    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:40.834766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:40.834718    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:42.499983 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.499945    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:37:42.543077 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.542995    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-tls-certs\") pod \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") "
Sep 17 18:37:42.545172 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.545146    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" (UID: "57d71a0d-bcaa-4ed2-a09e-2d11cad877e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:42.643507 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.643478    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twdj7\" (UniqueName: \"kubernetes.io/projected/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-kube-api-access-twdj7\") pod \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\" (UID: \"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8\") "
Sep 17 18:37:42.643678 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.643600    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:42.645510 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.645481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-kube-api-access-twdj7" (OuterVolumeSpecName: "kube-api-access-twdj7") pod "57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" (UID: "57d71a0d-bcaa-4ed2-a09e-2d11cad877e8"). InnerVolumeSpecName "kube-api-access-twdj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:42.663610 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.663585    2561 generic.go:356] "Generic (PLEG): container finished" podID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerID="4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b" exitCode=0
Sep 17 18:37:42.663729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.663650    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"
Sep 17 18:37:42.663729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.663666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" event={"ID":"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8","Type":"ContainerDied","Data":"4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b"}
Sep 17 18:37:42.663729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.663715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l" event={"ID":"57d71a0d-bcaa-4ed2-a09e-2d11cad877e8","Type":"ContainerDied","Data":"3ab0a1f6a48884079c438ece7b3fd34636999c08ac643205053bd7ef0e41adc7"}
Sep 17 18:37:42.663838 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.663737    2561 scope.go:117] "RemoveContainer" containerID="4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b"
Sep 17 18:37:42.675078 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.675041    2561 scope.go:117] "RemoveContainer" containerID="4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b"
Sep 17 18:37:42.675469 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:37:42.675441    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b\": container with ID starting with 4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b not found: ID does not exist" containerID="4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b"
Sep 17 18:37:42.675562 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.675479    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b"} err="failed to get container status \"4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b\": rpc error: code = NotFound desc = could not find container \"4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b\": container with ID starting with 4fc5364a03af5b31aca640da48dfbdebcca15b6bdb4d08093b807672e8fbca8b not found: ID does not exist"
Sep 17 18:37:42.689058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.689032    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"]
Sep 17 18:37:42.692822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.692802    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45z258l"]
Sep 17 18:37:42.744278 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:42.744250    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdj7\" (UniqueName: \"kubernetes.io/projected/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8-kube-api-access-twdj7\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:43.021228 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:43.021195    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" path="/var/lib/kubelet/pods/57d71a0d-bcaa-4ed2-a09e-2d11cad877e8/volumes"
Sep 17 18:37:52.159129 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:52.159093    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"]
Sep 17 18:37:52.159711 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:52.159322    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy" containerID="cri-o://f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68" gracePeriod=30
Sep 17 18:37:57.406449 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.406420    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:37:57.474296 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-podinfo\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474466 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-socket\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474466 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474350    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-credential-socket\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474466 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-token\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474466 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474447    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istiod-ca-cert\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474726 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474470    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-envoy\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474726 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-data\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474726 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bblld\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-kube-api-access-bblld\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474726 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474546    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-certs\") pod \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\" (UID: \"4bcefdce-b19a-40df-bcab-3963b1b1c2b8\") "
Sep 17 18:37:57.474726 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474705    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:57.474993 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.474722    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:57.475051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.475009    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.475051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.475033    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-credential-socket\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.475255 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.475223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:57.475412 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.475381    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:57.475509 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.475467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:57.477097 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.477071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:37:57.477221 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.477122    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-kube-api-access-bblld" (OuterVolumeSpecName: "kube-api-access-bblld") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "kube-api-access-bblld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:57.477402 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.477377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:57.477402 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.477381    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4bcefdce-b19a-40df-bcab-3963b1b1c2b8" (UID: "4bcefdce-b19a-40df-bcab-3963b1b1c2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:57.575999 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.575970    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-token\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.575999 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.575992    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istiod-ca-cert\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.575999 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.576002    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-envoy\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.576217 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.576010    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-data\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.576217 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.576018    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblld\" (UniqueName: \"kubernetes.io/projected/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-kube-api-access-bblld\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.576217 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.576027    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-workload-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.576217 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.576036    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bcefdce-b19a-40df-bcab-3963b1b1c2b8-istio-podinfo\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:37:57.731950 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.731916    2561 generic.go:356] "Generic (PLEG): container finished" podID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerID="f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68" exitCode=0
Sep 17 18:37:57.732205 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.731962    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"
Sep 17 18:37:57.732205 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.731999    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" event={"ID":"4bcefdce-b19a-40df-bcab-3963b1b1c2b8","Type":"ContainerDied","Data":"f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68"}
Sep 17 18:37:57.732205 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.732052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6" event={"ID":"4bcefdce-b19a-40df-bcab-3963b1b1c2b8","Type":"ContainerDied","Data":"f86bfa64d6cfedc6eee5bc1957f7a83baafc12b31e00dce361c701966fd2c25e"}
Sep 17 18:37:57.732205 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.732080    2561 scope.go:117] "RemoveContainer" containerID="f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68"
Sep 17 18:37:57.741848 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.741831    2561 scope.go:117] "RemoveContainer" containerID="cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"
Sep 17 18:37:57.749452 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.749437    2561 scope.go:117] "RemoveContainer" containerID="f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68"
Sep 17 18:37:57.749727 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:37:57.749708    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68\": container with ID starting with f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68 not found: ID does not exist" containerID="f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68"
Sep 17 18:37:57.749773 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.749735    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68"} err="failed to get container status \"f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68\": rpc error: code = NotFound desc = could not find container \"f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68\": container with ID starting with f21c9500863a48e602ff06e591e81588cc357743ba71125b49427c0a469a7b68 not found: ID does not exist"
Sep 17 18:37:57.749773 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.749753    2561 scope.go:117] "RemoveContainer" containerID="cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"
Sep 17 18:37:57.750018 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:37:57.750002    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2\": container with ID starting with cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2 not found: ID does not exist" containerID="cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"
Sep 17 18:37:57.750055 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.750025    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2"} err="failed to get container status \"cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2\": rpc error: code = NotFound desc = could not find container \"cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2\": container with ID starting with cae9a8d5828997a7c90297ed4d5201e1d8cd73c4ac919df90d114e6a8aace1e2 not found: ID does not exist"
Sep 17 18:37:57.756769 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.756745    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"]
Sep 17 18:37:57.760965 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:57.760944    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-hvgm6"]
Sep 17 18:37:59.019886 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:37:59.019850    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" path="/var/lib/kubelet/pods/4bcefdce-b19a-40df-bcab-3963b1b1c2b8/volumes"
Sep 17 18:39:14.406364 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406328    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"]
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406731    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406745    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406757    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406763    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406776    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406781    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406854    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy"
Sep 17 18:39:14.406873 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.406861    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d71a0d-bcaa-4ed2-a09e-2d11cad877e8" containerName="main"
Sep 17 18:39:14.409780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.409760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.412252 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.412227    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:39:14.412382 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.412230    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ml45c\""
Sep 17 18:39:14.412455 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.412375    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 18:39:14.412455 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.412401    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:39:14.420795 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.420775    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"]
Sep 17 18:39:14.532373 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.532339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba537d7-cea4-4db5-9444-a13114e6af57-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.532558 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.532377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.532558 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.532423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.532558 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.532457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.532558 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.532495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqqn\" (UniqueName: \"kubernetes.io/projected/bba537d7-cea4-4db5-9444-a13114e6af57-kube-api-access-tjqqn\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.532558 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.532541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633320 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba537d7-cea4-4db5-9444-a13114e6af57-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633320 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633581 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633581 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633581 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqqn\" (UniqueName: \"kubernetes.io/projected/bba537d7-cea4-4db5-9444-a13114e6af57-kube-api-access-tjqqn\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633581 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633828 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633787    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.633828 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.633802    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.634044 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.634022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.635961 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.635940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.636157 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.636138    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba537d7-cea4-4db5-9444-a13114e6af57-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:14.641732 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:14.641708    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqqn\" (UniqueName: \"kubernetes.io/projected/bba537d7-cea4-4db5-9444-a13114e6af57-kube-api-access-tjqqn\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:17.864815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:17.864775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:17.871515 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:17.871469    2561 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:17.965949 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:17.965913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:17.966106 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:17.966072    2561 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:17.968810 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:17.968786    2561 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") DevicePath \"csi-ee98ee1766dc0e4f78b1a3f1e518c03b4b00caaae4bee898d846d21a4a373c61\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:18.064387 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:18.064357    2561 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/66f33fb4ae41959ba0894d927c823da785849effc116e74f6244a4cb88c5813f/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:18.084091 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:18.084061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:18.320695 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:18.320653    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:18.598698 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:18.598669    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"]
Sep 17 18:39:18.600387 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:39:18.600361    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba537d7_cea4_4db5_9444_a13114e6af57.slice/crio-08569fe59bbe146922200822d64b9e5eba214715b8a3611100f24c403085c5fe WatchSource:0}: Error finding container 08569fe59bbe146922200822d64b9e5eba214715b8a3611100f24c403085c5fe: Status 404 returned error can't find the container with id 08569fe59bbe146922200822d64b9e5eba214715b8a3611100f24c403085c5fe
Sep 17 18:39:19.097368 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:19.097333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" event={"ID":"bba537d7-cea4-4db5-9444-a13114e6af57","Type":"ContainerStarted","Data":"070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428"}
Sep 17 18:39:19.097796 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:19.097392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" event={"ID":"bba537d7-cea4-4db5-9444-a13114e6af57","Type":"ContainerStarted","Data":"08569fe59bbe146922200822d64b9e5eba214715b8a3611100f24c403085c5fe"}
Sep 17 18:39:19.121217 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:19.121153    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podStartSLOduration=5.121127203 podStartE2EDuration="5.121127203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:19.119047489 +0000 UTC m=+3884.078932760" watchObservedRunningTime="2026-09-17 18:39:19.121127203 +0000 UTC m=+3884.081012473"
Sep 17 18:39:19.321483 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:19.321426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:28.321815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:28.321767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:39:28.324203 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:28.323483    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:39:37.264032 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:37.264001    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:39:37.272659 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:37.272619    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:39:38.321819 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:38.321752    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:39:48.321757 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:48.321701    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:39:58.321382 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:39:58.321340    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:40:08.321447 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:40:08.321361    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:40:18.321868 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:40:18.321818    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:40:28.321757 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:40:28.321660    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:40:38.331044 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:40:38.331003    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:40:38.339055 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:40:38.339031    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:41:15.793189 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:15.793157    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5_bba537d7-cea4-4db5-9444-a13114e6af57/main/0.log"
Sep 17 18:41:16.824726 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:16.824693    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"]
Sep 17 18:41:31.846863 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:31.846802    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" containerID="cri-o://070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428" gracePeriod=45
Sep 17 18:41:32.332499 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.332461    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 17 18:41:32.487209 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.487183    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:41:32.604971 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.604893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-home\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.604971 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.604935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba537d7-cea4-4db5-9444-a13114e6af57-tls-certs\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.604971 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.604959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjqqn\" (UniqueName: \"kubernetes.io/projected/bba537d7-cea4-4db5-9444-a13114e6af57-kube-api-access-tjqqn\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.605228 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-tmp-dir\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.605228 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605025    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-dshm\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.605228 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605077    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-model-cache\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.605228 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605171    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"bba537d7-cea4-4db5-9444-a13114e6af57\" (UID: \"bba537d7-cea4-4db5-9444-a13114e6af57\") "
Sep 17 18:41:32.605460 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605288    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-home" (OuterVolumeSpecName: "home") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:32.605460 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605438    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:41:32.605613 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.605585    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-model-cache" (OuterVolumeSpecName: "model-cache") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:32.607304 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.607255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-dshm" (OuterVolumeSpecName: "dshm") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:32.607407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.607339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba537d7-cea4-4db5-9444-a13114e6af57-kube-api-access-tjqqn" (OuterVolumeSpecName: "kube-api-access-tjqqn") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "kube-api-access-tjqqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:32.607524 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.607490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba537d7-cea4-4db5-9444-a13114e6af57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:32.613777 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.613753    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe" (OuterVolumeSpecName: "kserve-pvc-source") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:41:32.616552 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.616529    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bba537d7-cea4-4db5-9444-a13114e6af57" (UID: "bba537d7-cea4-4db5-9444-a13114e6af57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:32.706911 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.706885    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bba537d7-cea4-4db5-9444-a13114e6af57-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:41:32.706911 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.706909    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjqqn\" (UniqueName: \"kubernetes.io/projected/bba537d7-cea4-4db5-9444-a13114e6af57-kube-api-access-tjqqn\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:41:32.707058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.706919    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:41:32.707058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.706929    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:41:32.707058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.706936    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bba537d7-cea4-4db5-9444-a13114e6af57-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:41:32.707058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.706963    2561 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") on node \"ip-10-0-141-122.ec2.internal\" "
Sep 17 18:41:32.735579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.735561    2561 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe") on node "ip-10-0-141-122.ec2.internal"
Sep 17 18:41:32.765696 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.765668    2561 generic.go:356] "Generic (PLEG): container finished" podID="bba537d7-cea4-4db5-9444-a13114e6af57" containerID="070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428" exitCode=0
Sep 17 18:41:32.765799 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.765727    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"
Sep 17 18:41:32.765799 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.765751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" event={"ID":"bba537d7-cea4-4db5-9444-a13114e6af57","Type":"ContainerDied","Data":"070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428"}
Sep 17 18:41:32.765877 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.765825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5" event={"ID":"bba537d7-cea4-4db5-9444-a13114e6af57","Type":"ContainerDied","Data":"08569fe59bbe146922200822d64b9e5eba214715b8a3611100f24c403085c5fe"}
Sep 17 18:41:32.765877 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.765851    2561 scope.go:117] "RemoveContainer" containerID="070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428"
Sep 17 18:41:32.774816 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.774800    2561 scope.go:117] "RemoveContainer" containerID="070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428"
Sep 17 18:41:32.775161 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:41:32.775140    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428\": container with ID starting with 070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428 not found: ID does not exist" containerID="070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428"
Sep 17 18:41:32.775240 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.775168    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428"} err="failed to get container status \"070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428\": rpc error: code = NotFound desc = could not find container \"070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428\": container with ID starting with 070f848d82bfc3053625e0aef5c586ccf851c6424c6ee34ea89e5e1f93a5d428 not found: ID does not exist"
Sep 17 18:41:32.787475 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.787453    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"]
Sep 17 18:41:32.791717 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.791695    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-bl4j5"]
Sep 17 18:41:32.808345 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:32.808326    2561 reconciler_common.go:299] "Volume detached for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"csi-ee98ee1766dc0e4f78b1a3f1e518c03b4b00caaae4bee898d846d21a4a373c61\""
Sep 17 18:41:33.019846 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:33.019818    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" path="/var/lib/kubelet/pods/bba537d7-cea4-4db5-9444-a13114e6af57/volumes"
Sep 17 18:41:39.156766 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.156736    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"]
Sep 17 18:41:39.157179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.157043    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main"
Sep 17 18:41:39.157179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.157055    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main"
Sep 17 18:41:39.157179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.157134    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba537d7-cea4-4db5-9444-a13114e6af57" containerName="main"
Sep 17 18:41:39.157179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.157142    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bcefdce-b19a-40df-bcab-3963b1b1c2b8" containerName="istio-proxy"
Sep 17 18:41:39.161595 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.161575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.165391 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.165365    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:41:39.165526 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.165407    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:41:39.165526 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.165436    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-rr6g5\""
Sep 17 18:41:39.165526 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.165505    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-v4scn\""
Sep 17 18:41:39.165773 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.165512    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:41:39.169959 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.169940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"]
Sep 17 18:41:39.268051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.268018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046ee148-725f-415a-8ede-fefe4f1c234f-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.268196 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.268079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstjs\" (UniqueName: \"kubernetes.io/projected/046ee148-725f-415a-8ede-fefe4f1c234f-kube-api-access-rstjs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.368586 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.368547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstjs\" (UniqueName: \"kubernetes.io/projected/046ee148-725f-415a-8ede-fefe4f1c234f-kube-api-access-rstjs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.368805 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.368615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046ee148-725f-415a-8ede-fefe4f1c234f-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.371072 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.371053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046ee148-725f-415a-8ede-fefe4f1c234f-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.376823 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.376803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstjs\" (UniqueName: \"kubernetes.io/projected/046ee148-725f-415a-8ede-fefe4f1c234f-kube-api-access-rstjs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.472515 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.472430    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:39.748600 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.748571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"]
Sep 17 18:41:39.749925 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:41:39.749898    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046ee148_725f_415a_8ede_fefe4f1c234f.slice/crio-212cb49238e6dd2f2487f731a187cc2a8c99982b7b666874a8bc00fdef6dfcab WatchSource:0}: Error finding container 212cb49238e6dd2f2487f731a187cc2a8c99982b7b666874a8bc00fdef6dfcab: Status 404 returned error can't find the container with id 212cb49238e6dd2f2487f731a187cc2a8c99982b7b666874a8bc00fdef6dfcab
Sep 17 18:41:39.801371 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:39.801335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" event={"ID":"046ee148-725f-415a-8ede-fefe4f1c234f","Type":"ContainerStarted","Data":"212cb49238e6dd2f2487f731a187cc2a8c99982b7b666874a8bc00fdef6dfcab"}
Sep 17 18:41:40.807521 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:40.807481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" event={"ID":"046ee148-725f-415a-8ede-fefe4f1c234f","Type":"ContainerStarted","Data":"a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039"}
Sep 17 18:41:40.807949 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:40.807623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:41:40.825516 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:41:40.825458    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" podStartSLOduration=1.8254365670000001 podStartE2EDuration="1.825436567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:40.824793139 +0000 UTC m=+4025.784678409" watchObservedRunningTime="2026-09-17 18:41:40.825436567 +0000 UTC m=+4025.785321839"
Sep 17 18:42:11.812704 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:42:11.812669    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:43:20.606492 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.606409    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"]
Sep 17 18:43:20.609711 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.609693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.613729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.613703    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 18:43:20.613729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.613730    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 18:43:20.613934 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.613706    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-kkpfg\""
Sep 17 18:43:20.613934 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.613774    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 18:43:20.619206 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.619179    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"]
Sep 17 18:43:20.723216 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.723181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.723216 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.723217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtz5\" (UniqueName: \"kubernetes.io/projected/45fd2e89-92be-4adb-add4-7157032826e0-kube-api-access-mxtz5\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.723413 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.723274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.723413 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.723306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.723413 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.723332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fd2e89-92be-4adb-add4-7157032826e0-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.723413 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.723370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.824595 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.824564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.824595 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.824597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtz5\" (UniqueName: \"kubernetes.io/projected/45fd2e89-92be-4adb-add4-7157032826e0-kube-api-access-mxtz5\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.824884 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.824626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.824884 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.824664    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.824884 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.824683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fd2e89-92be-4adb-add4-7157032826e0-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.824884 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.824715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.825173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.825089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.825173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.825103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.825173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.825161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.827024 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.827003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.827169 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.827151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fd2e89-92be-4adb-add4-7157032826e0-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.833432 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.833410    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtz5\" (UniqueName: \"kubernetes.io/projected/45fd2e89-92be-4adb-add4-7157032826e0-kube-api-access-mxtz5\") pod \"scheduler-inline-config-test-kserve-8655bc757-8hr7h\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:20.920405 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:20.920314    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:21.196658 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:21.196553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"]
Sep 17 18:43:21.201234 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:43:21.201200    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fd2e89_92be_4adb_add4_7157032826e0.slice/crio-20073973b227e55c4fc17e16ed21ceed2011d42b6261a0d81153e69d4fcb0c8c WatchSource:0}: Error finding container 20073973b227e55c4fc17e16ed21ceed2011d42b6261a0d81153e69d4fcb0c8c: Status 404 returned error can't find the container with id 20073973b227e55c4fc17e16ed21ceed2011d42b6261a0d81153e69d4fcb0c8c
Sep 17 18:43:21.257055 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:21.257023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" event={"ID":"45fd2e89-92be-4adb-add4-7157032826e0","Type":"ContainerStarted","Data":"20073973b227e55c4fc17e16ed21ceed2011d42b6261a0d81153e69d4fcb0c8c"}
Sep 17 18:43:22.264285 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:22.264243    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" event={"ID":"45fd2e89-92be-4adb-add4-7157032826e0","Type":"ContainerStarted","Data":"9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1"}
Sep 17 18:43:22.288096 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:22.288038    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" podStartSLOduration=2.288020769 podStartE2EDuration="2.288020769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:22.286904787 +0000 UTC m=+4127.246790057" watchObservedRunningTime="2026-09-17 18:43:22.288020769 +0000 UTC m=+4127.247906040"
Sep 17 18:43:22.920660 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:22.920607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:30.920875 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:30.920831    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:30.933490 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:30.933462    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:31.316288 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:31.316259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:44.076002 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:44.075968    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"]
Sep 17 18:43:51.813256 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:51.813218    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:54.928647 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:54.928599    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"]
Sep 17 18:43:54.939257 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:43:54.939223    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:43:54.939257 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:43:54.939257 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:43:54.939257 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1" cmd=["/bin/sleep","15"]
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:54.939270    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" message="" err=<
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:43:54.939283    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" podUID="45fd2e89-92be-4adb-add4-7157032826e0" containerName="main" containerID="cri-o://9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1"
Sep 17 18:43:54.939529 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:54.939313    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" podUID="45fd2e89-92be-4adb-add4-7157032826e0" containerName="main" containerID="cri-o://9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1" gracePeriod=60
Sep 17 18:43:55.305814 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.305769    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" podUID="45fd2e89-92be-4adb-add4-7157032826e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 18:43:55.429108 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.429075    2561 generic.go:356] "Generic (PLEG): container finished" podID="45fd2e89-92be-4adb-add4-7157032826e0" containerID="9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1" exitCode=0
Sep 17 18:43:55.429276 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.429147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" event={"ID":"45fd2e89-92be-4adb-add4-7157032826e0","Type":"ContainerDied","Data":"9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1"}
Sep 17 18:43:55.681049 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.681028    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:55.726408 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-dshm\") pod \"45fd2e89-92be-4adb-add4-7157032826e0\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") "
Sep 17 18:43:55.726579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-tmp-dir\") pod \"45fd2e89-92be-4adb-add4-7157032826e0\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") "
Sep 17 18:43:55.726579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726473    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fd2e89-92be-4adb-add4-7157032826e0-tls-certs\") pod \"45fd2e89-92be-4adb-add4-7157032826e0\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") "
Sep 17 18:43:55.726579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxtz5\" (UniqueName: \"kubernetes.io/projected/45fd2e89-92be-4adb-add4-7157032826e0-kube-api-access-mxtz5\") pod \"45fd2e89-92be-4adb-add4-7157032826e0\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") "
Sep 17 18:43:55.726579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726533    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-home\") pod \"45fd2e89-92be-4adb-add4-7157032826e0\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") "
Sep 17 18:43:55.726579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726563    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-model-cache\") pod \"45fd2e89-92be-4adb-add4-7157032826e0\" (UID: \"45fd2e89-92be-4adb-add4-7157032826e0\") "
Sep 17 18:43:55.726918 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45fd2e89-92be-4adb-add4-7157032826e0" (UID: "45fd2e89-92be-4adb-add4-7157032826e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:55.726918 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726834    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-home" (OuterVolumeSpecName: "home") pod "45fd2e89-92be-4adb-add4-7157032826e0" (UID: "45fd2e89-92be-4adb-add4-7157032826e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:55.726918 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.726877    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "45fd2e89-92be-4adb-add4-7157032826e0" (UID: "45fd2e89-92be-4adb-add4-7157032826e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:55.728900 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.728861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fd2e89-92be-4adb-add4-7157032826e0-kube-api-access-mxtz5" (OuterVolumeSpecName: "kube-api-access-mxtz5") pod "45fd2e89-92be-4adb-add4-7157032826e0" (UID: "45fd2e89-92be-4adb-add4-7157032826e0"). InnerVolumeSpecName "kube-api-access-mxtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:55.728900 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.728889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-dshm" (OuterVolumeSpecName: "dshm") pod "45fd2e89-92be-4adb-add4-7157032826e0" (UID: "45fd2e89-92be-4adb-add4-7157032826e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:55.729061 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.728955    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fd2e89-92be-4adb-add4-7157032826e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45fd2e89-92be-4adb-add4-7157032826e0" (UID: "45fd2e89-92be-4adb-add4-7157032826e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:55.827929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.827884    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtz5\" (UniqueName: \"kubernetes.io/projected/45fd2e89-92be-4adb-add4-7157032826e0-kube-api-access-mxtz5\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:43:55.827929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.827924    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:43:55.827929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.827934    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:43:55.827929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.827943    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:43:55.828203 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.827951    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fd2e89-92be-4adb-add4-7157032826e0-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:43:55.828203 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:55.827959    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fd2e89-92be-4adb-add4-7157032826e0-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:43:56.434359 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:56.434331    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"
Sep 17 18:43:56.434836 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:56.434377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h" event={"ID":"45fd2e89-92be-4adb-add4-7157032826e0","Type":"ContainerDied","Data":"20073973b227e55c4fc17e16ed21ceed2011d42b6261a0d81153e69d4fcb0c8c"}
Sep 17 18:43:56.434836 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:56.434437    2561 scope.go:117] "RemoveContainer" containerID="9f846d2fd51102655375658b7e369cbb8c54ef0d416c1c9c5c6eb1da75b17cc1"
Sep 17 18:43:56.455611 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:56.455587    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"]
Sep 17 18:43:56.459745 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:56.459719    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8hr7h"]
Sep 17 18:43:57.020852 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:57.020820    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fd2e89-92be-4adb-add4-7157032826e0" path="/var/lib/kubelet/pods/45fd2e89-92be-4adb-add4-7157032826e0/volumes"
Sep 17 18:43:59.095947 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:43:59.095907    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main" containerID="cri-o://a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039" gracePeriod=45
Sep 17 18:44:01.813301 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:01.813266    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:11.812760 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:11.812721    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:11.813131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:11.812830    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:44:21.813180 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:21.813142    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:29.345045 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.345005    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:44:29.519583 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.519554    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rstjs\" (UniqueName: \"kubernetes.io/projected/046ee148-725f-415a-8ede-fefe4f1c234f-kube-api-access-rstjs\") pod \"046ee148-725f-415a-8ede-fefe4f1c234f\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") "
Sep 17 18:44:29.519794 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.519640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046ee148-725f-415a-8ede-fefe4f1c234f-tls-certs\") pod \"046ee148-725f-415a-8ede-fefe4f1c234f\" (UID: \"046ee148-725f-415a-8ede-fefe4f1c234f\") "
Sep 17 18:44:29.521927 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.521895    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046ee148-725f-415a-8ede-fefe4f1c234f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "046ee148-725f-415a-8ede-fefe4f1c234f" (UID: "046ee148-725f-415a-8ede-fefe4f1c234f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:29.522053 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.521907    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046ee148-725f-415a-8ede-fefe4f1c234f-kube-api-access-rstjs" (OuterVolumeSpecName: "kube-api-access-rstjs") pod "046ee148-725f-415a-8ede-fefe4f1c234f" (UID: "046ee148-725f-415a-8ede-fefe4f1c234f"). InnerVolumeSpecName "kube-api-access-rstjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:29.589414 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.589386    2561 generic.go:356] "Generic (PLEG): container finished" podID="046ee148-725f-415a-8ede-fefe4f1c234f" containerID="a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039" exitCode=0
Sep 17 18:44:29.589546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.589428    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"
Sep 17 18:44:29.589546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.589438    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" event={"ID":"046ee148-725f-415a-8ede-fefe4f1c234f","Type":"ContainerDied","Data":"a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039"}
Sep 17 18:44:29.589546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.589483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849" event={"ID":"046ee148-725f-415a-8ede-fefe4f1c234f","Type":"ContainerDied","Data":"212cb49238e6dd2f2487f731a187cc2a8c99982b7b666874a8bc00fdef6dfcab"}
Sep 17 18:44:29.589546 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.589504    2561 scope.go:117] "RemoveContainer" containerID="a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039"
Sep 17 18:44:29.598161 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.598144    2561 scope.go:117] "RemoveContainer" containerID="a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039"
Sep 17 18:44:29.598414 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:44:29.598396    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039\": container with ID starting with a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039 not found: ID does not exist" containerID="a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039"
Sep 17 18:44:29.598459 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.598419    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039"} err="failed to get container status \"a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039\": rpc error: code = NotFound desc = could not find container \"a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039\": container with ID starting with a14b5df03f1055dd8e4b4581746f2c2f70a21402ff6fe555f65c919670f80039 not found: ID does not exist"
Sep 17 18:44:29.610727 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.610707    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"]
Sep 17 18:44:29.613741 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.613722    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-9h849"]
Sep 17 18:44:29.620494 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.620477    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046ee148-725f-415a-8ede-fefe4f1c234f-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:44:29.620559 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:29.620497    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstjs\" (UniqueName: \"kubernetes.io/projected/046ee148-725f-415a-8ede-fefe4f1c234f-kube-api-access-rstjs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:44:31.020298 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:31.020262    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" path="/var/lib/kubelet/pods/046ee148-725f-415a-8ede-fefe4f1c234f/volumes"
Sep 17 18:44:37.292272 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:37.292248    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:44:37.300540 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:37.300516    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:44:45.833493 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833460    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"]
Sep 17 18:44:45.833899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fd2e89-92be-4adb-add4-7157032826e0" containerName="main"
Sep 17 18:44:45.833899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833807    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fd2e89-92be-4adb-add4-7157032826e0" containerName="main"
Sep 17 18:44:45.833899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833825    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main"
Sep 17 18:44:45.833899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833831    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main"
Sep 17 18:44:45.833899 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833902    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fd2e89-92be-4adb-add4-7157032826e0" containerName="main"
Sep 17 18:44:45.834058 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.833908    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="046ee148-725f-415a-8ede-fefe4f1c234f" containerName="main"
Sep 17 18:44:45.836792 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.836769    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:45.840407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.840365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-hb4bg\""
Sep 17 18:44:45.840407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.840399    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:44:45.840663 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.840419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-65nmv\""
Sep 17 18:44:45.840663 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.840437    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:44:45.840663 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.840401    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:44:45.849374 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.849351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"]
Sep 17 18:44:45.956755 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.956711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62wmx\" (UniqueName: \"kubernetes.io/projected/ff9792ce-aa89-4116-937c-ba47e00b4153-kube-api-access-62wmx\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:45.956929 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:45.956800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9792ce-aa89-4116-937c-ba47e00b4153-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.057963 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.057929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9792ce-aa89-4116-937c-ba47e00b4153-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.058141 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.057996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62wmx\" (UniqueName: \"kubernetes.io/projected/ff9792ce-aa89-4116-937c-ba47e00b4153-kube-api-access-62wmx\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.060847 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.060813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9792ce-aa89-4116-937c-ba47e00b4153-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.067050 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.067027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62wmx\" (UniqueName: \"kubernetes.io/projected/ff9792ce-aa89-4116-937c-ba47e00b4153-kube-api-access-62wmx\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.147178 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.147105    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.421504 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.421422    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"]
Sep 17 18:44:46.425172 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:44:46.425141    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9792ce_aa89_4116_937c_ba47e00b4153.slice/crio-5f8fb81991be0bcaf9cdd972fa6087e0fe801334e6f30403d86c212b47ff4f7f WatchSource:0}: Error finding container 5f8fb81991be0bcaf9cdd972fa6087e0fe801334e6f30403d86c212b47ff4f7f: Status 404 returned error can't find the container with id 5f8fb81991be0bcaf9cdd972fa6087e0fe801334e6f30403d86c212b47ff4f7f
Sep 17 18:44:46.671303 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.671263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" event={"ID":"ff9792ce-aa89-4116-937c-ba47e00b4153","Type":"ContainerStarted","Data":"2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d"}
Sep 17 18:44:46.671453 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.671324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" event={"ID":"ff9792ce-aa89-4116-937c-ba47e00b4153","Type":"ContainerStarted","Data":"5f8fb81991be0bcaf9cdd972fa6087e0fe801334e6f30403d86c212b47ff4f7f"}
Sep 17 18:44:46.671453 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.671351    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:44:46.692166 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:46.692055    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podStartSLOduration=1.692033552 podStartE2EDuration="1.692033552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:46.689974861 +0000 UTC m=+4211.649860131" watchObservedRunningTime="2026-09-17 18:44:46.692033552 +0000 UTC m=+4211.651918824"
Sep 17 18:44:56.893686 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.893595    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"]
Sep 17 18:44:56.897262 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.897236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:56.900968 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.900941    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-kgshk\""
Sep 17 18:44:56.901113 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.900987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 18:44:56.901113 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.901004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 18:44:56.901113 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.900950    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 18:44:56.906594 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.906573    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"]
Sep 17 18:44:56.952410 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.952383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:56.952530 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.952492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:56.952530 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.952518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:56.952671 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.952604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:56.952729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.952671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn728\" (UniqueName: \"kubernetes.io/projected/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-kube-api-access-gn728\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:56.952729 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:56.952709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053114 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053114 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053304 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053304 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn728\" (UniqueName: \"kubernetes.io/projected/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-kube-api-access-gn728\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053304 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053304 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053242    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053520 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.053667 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.053586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.055594 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.055571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.055842 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.055824    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.060908 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.060888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn728\" (UniqueName: \"kubernetes.io/projected/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-kube-api-access-gn728\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.209763 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.209676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:44:57.480344 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.480323    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"]
Sep 17 18:44:57.483948 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:44:57.483917    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4dbee2_d6e5_4e17_b95d_c32c36529f70.slice/crio-4bff6d5699c4b69c6c155f3b7b70b6b531487d8199b8dd7c96af46a2f45330a5 WatchSource:0}: Error finding container 4bff6d5699c4b69c6c155f3b7b70b6b531487d8199b8dd7c96af46a2f45330a5: Status 404 returned error can't find the container with id 4bff6d5699c4b69c6c155f3b7b70b6b531487d8199b8dd7c96af46a2f45330a5
Sep 17 18:44:57.723335 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.723291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" event={"ID":"6b4dbee2-d6e5-4e17-b95d-c32c36529f70","Type":"ContainerStarted","Data":"a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"}
Sep 17 18:44:57.723511 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:57.723366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" event={"ID":"6b4dbee2-d6e5-4e17-b95d-c32c36529f70","Type":"ContainerStarted","Data":"4bff6d5699c4b69c6c155f3b7b70b6b531487d8199b8dd7c96af46a2f45330a5"}
Sep 17 18:44:58.210429 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:44:58.210394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:45:07.210111 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:07.210072    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:45:07.222945 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:07.222916    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:45:07.241795 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:07.241751    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" podStartSLOduration=11.241735199 podStartE2EDuration="11.241735199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:57.744104886 +0000 UTC m=+4222.703990156" watchObservedRunningTime="2026-09-17 18:45:07.241735199 +0000 UTC m=+4232.201620510"
Sep 17 18:45:07.778579 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:07.778546    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:45:17.678398 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:17.678366    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:45:30.233469 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.233436    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"]
Sep 17 18:45:30.242716 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:45:30.242683    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:45:30.242716 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:30.242716 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:30.242716 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47" cmd=["/bin/sleep","15"]
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.242737    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" message="" err=<
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:45:30.242752    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" podUID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" containerName="main" containerID="cri-o://a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"
Sep 17 18:45:30.242982 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.242798    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" podUID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" containerName="main" containerID="cri-o://a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47" gracePeriod=60
Sep 17 18:45:30.589489 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.589464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:45:30.635055 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-home\") pod \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") "
Sep 17 18:45:30.635195 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tmp-dir\") pod \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") "
Sep 17 18:45:30.635195 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635106    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn728\" (UniqueName: \"kubernetes.io/projected/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-kube-api-access-gn728\") pod \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") "
Sep 17 18:45:30.635195 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635125    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tls-certs\") pod \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") "
Sep 17 18:45:30.635195 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635144    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-dshm\") pod \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") "
Sep 17 18:45:30.635363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-model-cache\") pod \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\" (UID: \"6b4dbee2-d6e5-4e17-b95d-c32c36529f70\") "
Sep 17 18:45:30.635363 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635324    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-home" (OuterVolumeSpecName: "home") pod "6b4dbee2-d6e5-4e17-b95d-c32c36529f70" (UID: "6b4dbee2-d6e5-4e17-b95d-c32c36529f70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:30.635443 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635364    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b4dbee2-d6e5-4e17-b95d-c32c36529f70" (UID: "6b4dbee2-d6e5-4e17-b95d-c32c36529f70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:30.635501 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635485    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:45:30.635568 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635502    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:45:30.635568 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.635533    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b4dbee2-d6e5-4e17-b95d-c32c36529f70" (UID: "6b4dbee2-d6e5-4e17-b95d-c32c36529f70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:30.637469 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.637441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-kube-api-access-gn728" (OuterVolumeSpecName: "kube-api-access-gn728") pod "6b4dbee2-d6e5-4e17-b95d-c32c36529f70" (UID: "6b4dbee2-d6e5-4e17-b95d-c32c36529f70"). InnerVolumeSpecName "kube-api-access-gn728". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:30.637469 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.637460    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-dshm" (OuterVolumeSpecName: "dshm") pod "6b4dbee2-d6e5-4e17-b95d-c32c36529f70" (UID: "6b4dbee2-d6e5-4e17-b95d-c32c36529f70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:30.637659 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.637467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b4dbee2-d6e5-4e17-b95d-c32c36529f70" (UID: "6b4dbee2-d6e5-4e17-b95d-c32c36529f70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:30.736404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.736371    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:45:30.736404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.736395    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:45:30.736404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.736407    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn728\" (UniqueName: \"kubernetes.io/projected/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-kube-api-access-gn728\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:45:30.736685 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.736416    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b4dbee2-d6e5-4e17-b95d-c32c36529f70-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:45:30.876068 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.875991    2561 generic.go:356] "Generic (PLEG): container finished" podID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" containerID="a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47" exitCode=0
Sep 17 18:45:30.876068 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.876038    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" event={"ID":"6b4dbee2-d6e5-4e17-b95d-c32c36529f70","Type":"ContainerDied","Data":"a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"}
Sep 17 18:45:30.876247 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.876074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh" event={"ID":"6b4dbee2-d6e5-4e17-b95d-c32c36529f70","Type":"ContainerDied","Data":"4bff6d5699c4b69c6c155f3b7b70b6b531487d8199b8dd7c96af46a2f45330a5"}
Sep 17 18:45:30.876247 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.876095    2561 scope.go:117] "RemoveContainer" containerID="a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"
Sep 17 18:45:30.876247 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.876038    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"
Sep 17 18:45:30.884955 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.884935    2561 scope.go:117] "RemoveContainer" containerID="a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"
Sep 17 18:45:30.885208 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:45:30.885187    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47\": container with ID starting with a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47 not found: ID does not exist" containerID="a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"
Sep 17 18:45:30.885275 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.885221    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47"} err="failed to get container status \"a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47\": rpc error: code = NotFound desc = could not find container \"a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47\": container with ID starting with a9617d4489d0150cdc72cd9ce273022047ab2a2e647ed918523d3fca0b5b9a47 not found: ID does not exist"
Sep 17 18:45:30.897306 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.897256    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"]
Sep 17 18:45:30.899568 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:30.899548    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f9bjjh"]
Sep 17 18:45:31.020704 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:45:31.020678    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" path="/var/lib/kubelet/pods/6b4dbee2-d6e5-4e17-b95d-c32c36529f70/volumes"
Sep 17 18:46:41.806705 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.806582    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"]
Sep 17 18:46:41.807302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.807109    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" containerName="main"
Sep 17 18:46:41.807302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.807130    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" containerName="main"
Sep 17 18:46:41.807302 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.807250    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4dbee2-d6e5-4e17-b95d-c32c36529f70" containerName="main"
Sep 17 18:46:41.810272 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.810252    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:41.813986 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.813966    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 18:46:41.814119 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.814025    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 18:46:41.814119 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.814030    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-ffzf7\""
Sep 17 18:46:41.814119 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.814031    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 18:46:41.818745 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.818725    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"]
Sep 17 18:46:41.938021 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.937991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:41.938021 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.938023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:41.938220 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.938043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee974eb-5a40-4620-9f52-43257222b698-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:41.938220 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.938102    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:41.938220 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.938164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:41.938220 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:41.938212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvptk\" (UniqueName: \"kubernetes.io/projected/fee974eb-5a40-4620-9f52-43257222b698-kube-api-access-lvptk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.038620 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.038577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.038620 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.038619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.038865 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.038650    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee974eb-5a40-4620-9f52-43257222b698-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.038865 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.038671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.038865 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.038696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.038865 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.038744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvptk\" (UniqueName: \"kubernetes.io/projected/fee974eb-5a40-4620-9f52-43257222b698-kube-api-access-lvptk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.039086 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.039009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.039147 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.039086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.039207 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.039161    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.041251 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.041216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.041589 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.041568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee974eb-5a40-4620-9f52-43257222b698-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.053623 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.053587    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvptk\" (UniqueName: \"kubernetes.io/projected/fee974eb-5a40-4620-9f52-43257222b698-kube-api-access-lvptk\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.121826 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.121745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:42.394946 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:42.394918    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"]
Sep 17 18:46:42.397533 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:46:42.397503    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee974eb_5a40_4620_9f52_43257222b698.slice/crio-8a84c50ce40296e54d8da7b72e4f7f8780b9a2390fd23a99ea33016a0e1a3a0c WatchSource:0}: Error finding container 8a84c50ce40296e54d8da7b72e4f7f8780b9a2390fd23a99ea33016a0e1a3a0c: Status 404 returned error can't find the container with id 8a84c50ce40296e54d8da7b72e4f7f8780b9a2390fd23a99ea33016a0e1a3a0c
Sep 17 18:46:43.193431 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:43.193398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" event={"ID":"fee974eb-5a40-4620-9f52-43257222b698","Type":"ContainerStarted","Data":"a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"}
Sep 17 18:46:43.193835 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:43.193450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" event={"ID":"fee974eb-5a40-4620-9f52-43257222b698","Type":"ContainerStarted","Data":"8a84c50ce40296e54d8da7b72e4f7f8780b9a2390fd23a99ea33016a0e1a3a0c"}
Sep 17 18:46:43.212917 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:43.212871    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" podStartSLOduration=2.21285582 podStartE2EDuration="2.21285582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:43.211294108 +0000 UTC m=+4328.171179378" watchObservedRunningTime="2026-09-17 18:46:43.21285582 +0000 UTC m=+4328.172741090"
Sep 17 18:46:44.122171 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:44.122138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:52.122552 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:52.122470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:52.134960 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:52.134932    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:46:52.245368 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:46:52.245339    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:47:13.869473 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:13.869434    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"]
Sep 17 18:47:17.184103 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.184065    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"]
Sep 17 18:47:17.194276 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:47:17.194237    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:47:17.194276 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:17.194276 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:17.194276 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566" cmd=["/bin/sleep","15"]
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.194300    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" message="" err=<
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:47:17.194318    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" podUID="fee974eb-5a40-4620-9f52-43257222b698" containerName="main" containerID="cri-o://a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"
Sep 17 18:47:17.194601 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.194360    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" podUID="fee974eb-5a40-4620-9f52-43257222b698" containerName="main" containerID="cri-o://a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566" gracePeriod=60
Sep 17 18:47:17.234646 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.234589    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" podUID="fee974eb-5a40-4620-9f52-43257222b698" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Sep 17 18:47:17.551304 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.551280    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:47:17.642583 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee974eb-5a40-4620-9f52-43257222b698-tls-certs\") pod \"fee974eb-5a40-4620-9f52-43257222b698\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") "
Sep 17 18:47:17.642815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642670    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvptk\" (UniqueName: \"kubernetes.io/projected/fee974eb-5a40-4620-9f52-43257222b698-kube-api-access-lvptk\") pod \"fee974eb-5a40-4620-9f52-43257222b698\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") "
Sep 17 18:47:17.642815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642704    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-home\") pod \"fee974eb-5a40-4620-9f52-43257222b698\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") "
Sep 17 18:47:17.642815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-tmp-dir\") pod \"fee974eb-5a40-4620-9f52-43257222b698\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") "
Sep 17 18:47:17.642815 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642812    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-model-cache\") pod \"fee974eb-5a40-4620-9f52-43257222b698\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") "
Sep 17 18:47:17.643038 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642847    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-dshm\") pod \"fee974eb-5a40-4620-9f52-43257222b698\" (UID: \"fee974eb-5a40-4620-9f52-43257222b698\") "
Sep 17 18:47:17.643038 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.642970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-home" (OuterVolumeSpecName: "home") pod "fee974eb-5a40-4620-9f52-43257222b698" (UID: "fee974eb-5a40-4620-9f52-43257222b698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:17.643153 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.643010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fee974eb-5a40-4620-9f52-43257222b698" (UID: "fee974eb-5a40-4620-9f52-43257222b698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:17.643153 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.643078    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-model-cache" (OuterVolumeSpecName: "model-cache") pod "fee974eb-5a40-4620-9f52-43257222b698" (UID: "fee974eb-5a40-4620-9f52-43257222b698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:17.643153 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.643147    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:17.643313 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.643164    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:17.645131 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.645090    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee974eb-5a40-4620-9f52-43257222b698-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fee974eb-5a40-4620-9f52-43257222b698" (UID: "fee974eb-5a40-4620-9f52-43257222b698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:17.645250 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.645198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee974eb-5a40-4620-9f52-43257222b698-kube-api-access-lvptk" (OuterVolumeSpecName: "kube-api-access-lvptk") pod "fee974eb-5a40-4620-9f52-43257222b698" (UID: "fee974eb-5a40-4620-9f52-43257222b698"). InnerVolumeSpecName "kube-api-access-lvptk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:17.645250 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.645196    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-dshm" (OuterVolumeSpecName: "dshm") pod "fee974eb-5a40-4620-9f52-43257222b698" (UID: "fee974eb-5a40-4620-9f52-43257222b698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:17.678756 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.678692    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:17.744211 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.744183    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:17.744211 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.744205    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fee974eb-5a40-4620-9f52-43257222b698-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:17.744211 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.744215    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fee974eb-5a40-4620-9f52-43257222b698-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:17.744402 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:17.744224    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvptk\" (UniqueName: \"kubernetes.io/projected/fee974eb-5a40-4620-9f52-43257222b698-kube-api-access-lvptk\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:18.362882 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.362848    2561 generic.go:356] "Generic (PLEG): container finished" podID="fee974eb-5a40-4620-9f52-43257222b698" containerID="a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566" exitCode=0
Sep 17 18:47:18.363273 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.362897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" event={"ID":"fee974eb-5a40-4620-9f52-43257222b698","Type":"ContainerDied","Data":"a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"}
Sep 17 18:47:18.363273 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.362917    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"
Sep 17 18:47:18.363273 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.362967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz" event={"ID":"fee974eb-5a40-4620-9f52-43257222b698","Type":"ContainerDied","Data":"8a84c50ce40296e54d8da7b72e4f7f8780b9a2390fd23a99ea33016a0e1a3a0c"}
Sep 17 18:47:18.363273 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.362991    2561 scope.go:117] "RemoveContainer" containerID="a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"
Sep 17 18:47:18.375454 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.375404    2561 scope.go:117] "RemoveContainer" containerID="a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"
Sep 17 18:47:18.375703 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:47:18.375683    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566\": container with ID starting with a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566 not found: ID does not exist" containerID="a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"
Sep 17 18:47:18.375780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.375710    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566"} err="failed to get container status \"a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566\": rpc error: code = NotFound desc = could not find container \"a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566\": container with ID starting with a67d28bc3ba1894f4af6a7486c04ac06d1444c7e64e51d32a1133c40fb3dd566 not found: ID does not exist"
Sep 17 18:47:18.386432 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.386412    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"]
Sep 17 18:47:18.392308 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:18.392288    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-zxprz"]
Sep 17 18:47:19.020965 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:19.020931    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee974eb-5a40-4620-9f52-43257222b698" path="/var/lib/kubelet/pods/fee974eb-5a40-4620-9f52-43257222b698/volumes"
Sep 17 18:47:27.678034 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:27.677998    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:28.889606 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:28.889559    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main" containerID="cri-o://2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d" gracePeriod=45
Sep 17 18:47:37.678821 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:37.678783    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:37.679179 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:37.678911    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:47:47.678428 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:47.678389    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:57.678559 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:57.678521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:59.132062 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.132037    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:47:59.183054 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.182985    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62wmx\" (UniqueName: \"kubernetes.io/projected/ff9792ce-aa89-4116-937c-ba47e00b4153-kube-api-access-62wmx\") pod \"ff9792ce-aa89-4116-937c-ba47e00b4153\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") "
Sep 17 18:47:59.183054 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.183041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9792ce-aa89-4116-937c-ba47e00b4153-tls-certs\") pod \"ff9792ce-aa89-4116-937c-ba47e00b4153\" (UID: \"ff9792ce-aa89-4116-937c-ba47e00b4153\") "
Sep 17 18:47:59.185084 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.185050    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9792ce-aa89-4116-937c-ba47e00b4153-kube-api-access-62wmx" (OuterVolumeSpecName: "kube-api-access-62wmx") pod "ff9792ce-aa89-4116-937c-ba47e00b4153" (UID: "ff9792ce-aa89-4116-937c-ba47e00b4153"). InnerVolumeSpecName "kube-api-access-62wmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:59.185192 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.185128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9792ce-aa89-4116-937c-ba47e00b4153-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff9792ce-aa89-4116-937c-ba47e00b4153" (UID: "ff9792ce-aa89-4116-937c-ba47e00b4153"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:59.287085 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.284313    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62wmx\" (UniqueName: \"kubernetes.io/projected/ff9792ce-aa89-4116-937c-ba47e00b4153-kube-api-access-62wmx\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:59.287085 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.284349    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff9792ce-aa89-4116-937c-ba47e00b4153-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:47:59.545146 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.545113    2561 generic.go:356] "Generic (PLEG): container finished" podID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerID="2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d" exitCode=0
Sep 17 18:47:59.545354 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.545187    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"
Sep 17 18:47:59.545354 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.545201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" event={"ID":"ff9792ce-aa89-4116-937c-ba47e00b4153","Type":"ContainerDied","Data":"2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d"}
Sep 17 18:47:59.545354 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.545271    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n" event={"ID":"ff9792ce-aa89-4116-937c-ba47e00b4153","Type":"ContainerDied","Data":"5f8fb81991be0bcaf9cdd972fa6087e0fe801334e6f30403d86c212b47ff4f7f"}
Sep 17 18:47:59.545354 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.545295    2561 scope.go:117] "RemoveContainer" containerID="2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d"
Sep 17 18:47:59.554541 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.554523    2561 scope.go:117] "RemoveContainer" containerID="2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d"
Sep 17 18:47:59.554873 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:47:59.554852    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d\": container with ID starting with 2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d not found: ID does not exist" containerID="2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d"
Sep 17 18:47:59.554945 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.554884    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d"} err="failed to get container status \"2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d\": rpc error: code = NotFound desc = could not find container \"2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d\": container with ID starting with 2885b90780b0c859075a1205ab0d0e57108e3a1e451675e799c6ba9d9b15512d not found: ID does not exist"
Sep 17 18:47:59.568421 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.568395    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"]
Sep 17 18:47:59.572066 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:47:59.572048    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-s5q6n"]
Sep 17 18:48:01.019824 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:01.019792    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" path="/var/lib/kubelet/pods/ff9792ce-aa89-4116-937c-ba47e00b4153/volumes"
Sep 17 18:48:42.405331 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.405296    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"]
Sep 17 18:48:42.405909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.405874    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main"
Sep 17 18:48:42.405909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.405895    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main"
Sep 17 18:48:42.405909 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.405907    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee974eb-5a40-4620-9f52-43257222b698" containerName="main"
Sep 17 18:48:42.406087 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.405916    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee974eb-5a40-4620-9f52-43257222b698" containerName="main"
Sep 17 18:48:42.406087 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.406016    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9792ce-aa89-4116-937c-ba47e00b4153" containerName="main"
Sep 17 18:48:42.406087 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.406029    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee974eb-5a40-4620-9f52-43257222b698" containerName="main"
Sep 17 18:48:42.410216 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.410195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.412645 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.412610    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 18:48:42.412770 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.412747    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-8ttfc\""
Sep 17 18:48:42.413664 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.413626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 18:48:42.413664 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.413659    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 18:48:42.418965 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.418943    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"]
Sep 17 18:48:42.531404 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.531363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls7k\" (UniqueName: \"kubernetes.io/projected/4d46568b-869e-4869-ba42-103f5a5128c4-kube-api-access-5ls7k\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.531572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.531418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.531572 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.531526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.531674 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.531588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.531674 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.531614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.531750 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.531696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d46568b-869e-4869-ba42-103f5a5128c4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.632995 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.632963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d46568b-869e-4869-ba42-103f5a5128c4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633173 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls7k\" (UniqueName: \"kubernetes.io/projected/4d46568b-869e-4869-ba42-103f5a5128c4-kube-api-access-5ls7k\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633460 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633522 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633498    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.633584 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.633539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.635606 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.635580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.635826 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.635804    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d46568b-869e-4869-ba42-103f5a5128c4-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.645421 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.645395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls7k\" (UniqueName: \"kubernetes.io/projected/4d46568b-869e-4869-ba42-103f5a5128c4-kube-api-access-5ls7k\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qx4tk\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.722654 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.722563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:42.997054 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:42.997023    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"]
Sep 17 18:48:42.998841 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:48:42.998802    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d46568b_869e_4869_ba42_103f5a5128c4.slice/crio-8b55a13d2d385d2c20ed4367f0637b899a8a68cf67c30cf153bcea096df2cecf WatchSource:0}: Error finding container 8b55a13d2d385d2c20ed4367f0637b899a8a68cf67c30cf153bcea096df2cecf: Status 404 returned error can't find the container with id 8b55a13d2d385d2c20ed4367f0637b899a8a68cf67c30cf153bcea096df2cecf
Sep 17 18:48:43.753436 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:43.753390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" event={"ID":"4d46568b-869e-4869-ba42-103f5a5128c4","Type":"ContainerStarted","Data":"ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"}
Sep 17 18:48:43.753979 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:43.753463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" event={"ID":"4d46568b-869e-4869-ba42-103f5a5128c4","Type":"ContainerStarted","Data":"8b55a13d2d385d2c20ed4367f0637b899a8a68cf67c30cf153bcea096df2cecf"}
Sep 17 18:48:43.773355 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:43.773306    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" podStartSLOduration=1.773288317 podStartE2EDuration="1.773288317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:43.771408601 +0000 UTC m=+4448.731293882" watchObservedRunningTime="2026-09-17 18:48:43.773288317 +0000 UTC m=+4448.733173587"
Sep 17 18:48:44.723487 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:44.723447    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:52.723798 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:52.723758    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:52.736396 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:52.736372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:48:52.810078 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:48:52.810054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:49:27.840623 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:27.840578    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"]
Sep 17 18:49:27.850125 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:49:27.850091    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:49:27.850125 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:27.850125 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:27.850125 ip-10-0-141-122 kubenswrapper[2561]:  > containerID="ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78" cmd=["/bin/sleep","15"]
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:27.850142    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" message="" err=<
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]:  >
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:49:27.850161    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" podUID="4d46568b-869e-4869-ba42-103f5a5128c4" containerName="main" containerID="cri-o://ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"
Sep 17 18:49:27.850407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:27.850226    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" podUID="4d46568b-869e-4869-ba42-103f5a5128c4" containerName="main" containerID="cri-o://ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78" gracePeriod=60
Sep 17 18:49:28.199807 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.199786    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:49:28.321646 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.321609    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-dshm\") pod \"4d46568b-869e-4869-ba42-103f5a5128c4\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") "
Sep 17 18:49:28.321869 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.321673    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d46568b-869e-4869-ba42-103f5a5128c4-tls-certs\") pod \"4d46568b-869e-4869-ba42-103f5a5128c4\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") "
Sep 17 18:49:28.321869 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.321698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-tmp-dir\") pod \"4d46568b-869e-4869-ba42-103f5a5128c4\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") "
Sep 17 18:49:28.321869 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.321727    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-model-cache\") pod \"4d46568b-869e-4869-ba42-103f5a5128c4\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") "
Sep 17 18:49:28.321869 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.321756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ls7k\" (UniqueName: \"kubernetes.io/projected/4d46568b-869e-4869-ba42-103f5a5128c4-kube-api-access-5ls7k\") pod \"4d46568b-869e-4869-ba42-103f5a5128c4\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") "
Sep 17 18:49:28.321869 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.321829    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-home\") pod \"4d46568b-869e-4869-ba42-103f5a5128c4\" (UID: \"4d46568b-869e-4869-ba42-103f5a5128c4\") "
Sep 17 18:49:28.322300 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.322017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d46568b-869e-4869-ba42-103f5a5128c4" (UID: "4d46568b-869e-4869-ba42-103f5a5128c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:28.322300 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.322071    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d46568b-869e-4869-ba42-103f5a5128c4" (UID: "4d46568b-869e-4869-ba42-103f5a5128c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:28.322300 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.322198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-home" (OuterVolumeSpecName: "home") pod "4d46568b-869e-4869-ba42-103f5a5128c4" (UID: "4d46568b-869e-4869-ba42-103f5a5128c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:28.323932 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.323907    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d46568b-869e-4869-ba42-103f5a5128c4-kube-api-access-5ls7k" (OuterVolumeSpecName: "kube-api-access-5ls7k") pod "4d46568b-869e-4869-ba42-103f5a5128c4" (UID: "4d46568b-869e-4869-ba42-103f5a5128c4"). InnerVolumeSpecName "kube-api-access-5ls7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:28.324120 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.324100    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d46568b-869e-4869-ba42-103f5a5128c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d46568b-869e-4869-ba42-103f5a5128c4" (UID: "4d46568b-869e-4869-ba42-103f5a5128c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:28.324120 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.324105    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-dshm" (OuterVolumeSpecName: "dshm") pod "4d46568b-869e-4869-ba42-103f5a5128c4" (UID: "4d46568b-869e-4869-ba42-103f5a5128c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:28.422596 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.422516    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-dshm\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:49:28.422596 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.422552    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d46568b-869e-4869-ba42-103f5a5128c4-tls-certs\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:49:28.422596 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.422561    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-tmp-dir\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:49:28.422596 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.422569    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-model-cache\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:49:28.422596 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.422577    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ls7k\" (UniqueName: \"kubernetes.io/projected/4d46568b-869e-4869-ba42-103f5a5128c4-kube-api-access-5ls7k\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:49:28.422596 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.422584    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d46568b-869e-4869-ba42-103f5a5128c4-home\") on node \"ip-10-0-141-122.ec2.internal\" DevicePath \"\""
Sep 17 18:49:28.961395 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.961364    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d46568b-869e-4869-ba42-103f5a5128c4" containerID="ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78" exitCode=0
Sep 17 18:49:28.961811 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.961408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" event={"ID":"4d46568b-869e-4869-ba42-103f5a5128c4","Type":"ContainerDied","Data":"ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"}
Sep 17 18:49:28.961811 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.961445    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk" event={"ID":"4d46568b-869e-4869-ba42-103f5a5128c4","Type":"ContainerDied","Data":"8b55a13d2d385d2c20ed4367f0637b899a8a68cf67c30cf153bcea096df2cecf"}
Sep 17 18:49:28.961811 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.961467    2561 scope.go:117] "RemoveContainer" containerID="ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"
Sep 17 18:49:28.961811 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.961412    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"
Sep 17 18:49:28.973269 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.973247    2561 scope.go:117] "RemoveContainer" containerID="ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"
Sep 17 18:49:28.973532 ip-10-0-141-122 kubenswrapper[2561]: E0917 18:49:28.973513    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78\": container with ID starting with ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78 not found: ID does not exist" containerID="ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"
Sep 17 18:49:28.973592 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.973540    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78"} err="failed to get container status \"ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78\": rpc error: code = NotFound desc = could not find container \"ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78\": container with ID starting with ae0ae50577e290f591b9eb033629b1ed6f366e6723908cf1b5a29d03b47c2e78 not found: ID does not exist"
Sep 17 18:49:28.985859 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.985835    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"]
Sep 17 18:49:28.990822 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:28.990799    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qx4tk"]
Sep 17 18:49:29.019523 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:29.019497    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d46568b-869e-4869-ba42-103f5a5128c4" path="/var/lib/kubelet/pods/4d46568b-869e-4869-ba42-103f5a5128c4/volumes"
Sep 17 18:49:37.319868 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:37.319840    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:49:37.328252 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:49:37.328228    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:50:53.642280 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:50:53.642199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/0.log"
Sep 17 18:50:53.644849 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:50:53.644830    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/1.log"
Sep 17 18:50:54.400998 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:50:54.400964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/0.log"
Sep 17 18:50:54.403335 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:50:54.403316    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/1.log"
Sep 17 18:50:55.214496 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:50:55.214470    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5zndp_16f07158-3254-45a8-8cee-e2637109b2b6/authorino/0.log"
Sep 17 18:50:55.230261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:50:55.230237    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-zkvgz_28ff21c9-b685-4e40-bf3c-5ca76fc32334/manager/0.log"
Sep 17 18:51:04.217791 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:04.217760    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d6dz7_d6c4029f-d3c5-4c0e-82fc-0f9f48968e8e/global-pull-secret-syncer/0.log"
Sep 17 18:51:04.340996 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:04.340953    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v5vhn_7d1e2c70-905d-42e5-a325-bb57bdb2edcd/konnectivity-agent/0.log"
Sep 17 18:51:04.364316 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:04.364292    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-122.ec2.internal_15339a5bbb7d25ed5fa56da3efddf311/haproxy/0.log"
Sep 17 18:51:09.393261 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:09.393177    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5zndp_16f07158-3254-45a8-8cee-e2637109b2b6/authorino/0.log"
Sep 17 18:51:09.443011 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:09.442984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-zkvgz_28ff21c9-b685-4e40-bf3c-5ca76fc32334/manager/0.log"
Sep 17 18:51:11.235516 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.235488    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c84f6d5c6-x2sw2_ca590260-399a-4d1e-a0bf-52744c163003/metrics-server/0.log"
Sep 17 18:51:11.263407 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.263380    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-q8z58_9a015893-311e-427b-b258-2fd5d85e6a25/monitoring-plugin/0.log"
Sep 17 18:51:11.296686 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.296667    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jv69z_28070dd8-00c3-4f8e-b471-17e7ee28253d/node-exporter/0.log"
Sep 17 18:51:11.324789 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.324769    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jv69z_28070dd8-00c3-4f8e-b471-17e7ee28253d/kube-rbac-proxy/0.log"
Sep 17 18:51:11.349741 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.349724    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jv69z_28070dd8-00c3-4f8e-b471-17e7ee28253d/init-textfile/0.log"
Sep 17 18:51:11.707051 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.707025    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/prometheus/0.log"
Sep 17 18:51:11.728248 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.728225    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/config-reloader/0.log"
Sep 17 18:51:11.758737 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.758708    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/thanos-sidecar/0.log"
Sep 17 18:51:11.785045 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.785023    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/kube-rbac-proxy-web/0.log"
Sep 17 18:51:11.814622 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.814596    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/kube-rbac-proxy/0.log"
Sep 17 18:51:11.837354 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.837332    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/kube-rbac-proxy-thanos/0.log"
Sep 17 18:51:11.869147 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.869131    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1155a300-f1e8-4cf6-9b79-fe4420a87288/init-config-reloader/0.log"
Sep 17 18:51:11.921690 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.921622    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-bwl9k_dd6b2f1d-6863-45b9-9f6d-29d791ad745a/prometheus-operator/0.log"
Sep 17 18:51:11.976948 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:11.976932    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-bwl9k_dd6b2f1d-6863-45b9-9f6d-29d791ad745a/kube-rbac-proxy/0.log"
Sep 17 18:51:12.030984 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:12.030962    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-2br2j_57dc139b-9e10-4290-869c-4190a55fedb8/prometheus-operator-admission-webhook/0.log"
Sep 17 18:51:12.083833 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:12.083810    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d689769bd-zw5n4_c101df20-1d1c-447d-8d81-dea041093e94/telemeter-client/0.log"
Sep 17 18:51:12.109151 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:12.109117    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d689769bd-zw5n4_c101df20-1d1c-447d-8d81-dea041093e94/reload/0.log"
Sep 17 18:51:12.133183 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:12.133163    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d689769bd-zw5n4_c101df20-1d1c-447d-8d81-dea041093e94/kube-rbac-proxy/0.log"
Sep 17 18:51:13.479691 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:13.479659    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-mg9vg_86261ca0-0197-4c31-8a44-462c460d2e01/networking-console-plugin/0.log"
Sep 17 18:51:13.977399 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:13.977323    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/2.log"
Sep 17 18:51:13.982096 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:13.982074    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-mjfcw_f2db503d-1499-4ea7-a841-ab983806a9f6/console-operator/3.log"
Sep 17 18:51:14.012894 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.012860    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"]
Sep 17 18:51:14.013205 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.013193    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d46568b-869e-4869-ba42-103f5a5128c4" containerName="main"
Sep 17 18:51:14.013253 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.013206    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d46568b-869e-4869-ba42-103f5a5128c4" containerName="main"
Sep 17 18:51:14.013288 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.013272    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d46568b-869e-4869-ba42-103f5a5128c4" containerName="main"
Sep 17 18:51:14.016833 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.016810    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.019197 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.019178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fdhq6\"/\"default-dockercfg-2qc86\""
Sep 17 18:51:14.019311 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.019180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fdhq6\"/\"kube-root-ca.crt\""
Sep 17 18:51:14.019311 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.019180    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fdhq6\"/\"openshift-service-ca.crt\""
Sep 17 18:51:14.024787 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.024766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"]
Sep 17 18:51:14.093287 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.093257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktdq\" (UniqueName: \"kubernetes.io/projected/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-kube-api-access-tktdq\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.093467 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.093307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-podres\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.093467 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.093357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-proc\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.093467 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.093402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-lib-modules\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.093467 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.093450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-sys\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194562 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194526    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-proc\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194767 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-lib-modules\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194767 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-sys\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194767 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktdq\" (UniqueName: \"kubernetes.io/projected/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-kube-api-access-tktdq\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194767 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-sys\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194767 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-proc\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194997 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-podres\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194997 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-lib-modules\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.194997 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.194835    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-podres\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.203733 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.203710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktdq\" (UniqueName: \"kubernetes.io/projected/3bcc514a-9619-417d-8d8f-dd5d5b4f9203-kube-api-access-tktdq\") pod \"perf-node-gather-daemonset-7jxdj\" (UID: \"3bcc514a-9619-417d-8d8f-dd5d5b4f9203\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.327702 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.327677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:14.595652 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:14.595612    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"]
Sep 17 18:51:14.598083 ip-10-0-141-122 kubenswrapper[2561]: W0917 18:51:14.598058    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3bcc514a_9619_417d_8d8f_dd5d5b4f9203.slice/crio-b5eaec8423bb8144e1168bf99a2f8115ca4b743671cc72d4cc83ae7283b34800 WatchSource:0}: Error finding container b5eaec8423bb8144e1168bf99a2f8115ca4b743671cc72d4cc83ae7283b34800: Status 404 returned error can't find the container with id b5eaec8423bb8144e1168bf99a2f8115ca4b743671cc72d4cc83ae7283b34800
Sep 17 18:51:15.438646 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:15.438593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj" event={"ID":"3bcc514a-9619-417d-8d8f-dd5d5b4f9203","Type":"ContainerStarted","Data":"f5e4a52a0bac192137fa7dcaae3b403b001ce07eada2e760b50221cafaacbcec"}
Sep 17 18:51:15.438850 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:15.438680    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:15.438850 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:15.438701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj" event={"ID":"3bcc514a-9619-417d-8d8f-dd5d5b4f9203","Type":"ContainerStarted","Data":"b5eaec8423bb8144e1168bf99a2f8115ca4b743671cc72d4cc83ae7283b34800"}
Sep 17 18:51:15.459925 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:15.459732    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj" podStartSLOduration=2.459711315 podStartE2EDuration="2.459711315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:15.458400559 +0000 UTC m=+4600.418285829" watchObservedRunningTime="2026-09-17 18:51:15.459711315 +0000 UTC m=+4600.419596590"
Sep 17 18:51:16.183642 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:16.183599    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vssft_cc9faa45-7dfc-4569-9c1f-30a623d562e7/dns/0.log"
Sep 17 18:51:16.206136 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:16.206110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vssft_cc9faa45-7dfc-4569-9c1f-30a623d562e7/kube-rbac-proxy/0.log"
Sep 17 18:51:16.257738 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:16.257715    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7qdqz_17dfa02f-bb9b-49b3-95d8-2a59a1e2f88b/dns-node-resolver/0.log"
Sep 17 18:51:16.863391 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:16.863361    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4jfq7_27603f15-420b-4c05-aaa8-fea5ecb33710/node-ca/0.log"
Sep 17 18:51:17.989113 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:17.989082    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/0.log"
Sep 17 18:51:17.991865 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:17.991843    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/1.log"
Sep 17 18:51:18.527557 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:18.527527    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bcllq_a94e2c20-8bbd-47d1-b3d1-391b109eccae/serve-healthcheck-canary/0.log"
Sep 17 18:51:19.018947 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:19.018917    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fjzlf_bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd/kube-rbac-proxy/0.log"
Sep 17 18:51:19.040174 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:19.040154    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fjzlf_bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd/exporter/0.log"
Sep 17 18:51:19.065370 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:19.065349    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fjzlf_bd1da701-bd7b-4c98-b0d1-f1f7bdf4e4cd/extractor/0.log"
Sep 17 18:51:21.453004 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:21.452975    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-7jxdj"
Sep 17 18:51:22.132479 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:22.132448    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6796bd7874-6hlb9_dac92b04-4493-4d46-9ae3-48550b88a6fb/manager/0.log"
Sep 17 18:51:22.185742 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:22.185718    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nk4ds_90930738-b7d8-42a2-8c32-eccd4a85be80/openshift-lws-operator/0.log"
Sep 17 18:51:23.172298 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:23.172275    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-vjldm_08d8a2cf-48f7-4919-8c75-1f688fa92368/server/0.log"
Sep 17 18:51:23.887340 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:23.887311    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/0.log"
Sep 17 18:51:23.889503 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:23.889483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/1.log"
Sep 17 18:51:24.994935 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:24.994909    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/0.log"
Sep 17 18:51:24.997524 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:24.997504    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-x4q7j_8bf9b8f0-b273-4f7a-8ee4-be8df6445e18/istio-proxy/1.log"
Sep 17 18:51:30.048900 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.048865    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8lsrh_52a1efed-a8d2-4761-bc6a-c180cda036d0/kube-multus/0.log"
Sep 17 18:51:30.428831 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.428757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:51:30.448443 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.448419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/egress-router-binary-copy/0.log"
Sep 17 18:51:30.467853 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.467822    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/cni-plugins/0.log"
Sep 17 18:51:30.490167 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.490143    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/bond-cni-plugin/0.log"
Sep 17 18:51:30.509780 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.509762    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/routeoverride-cni/0.log"
Sep 17 18:51:30.529070 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.529053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/whereabouts-cni-bincopy/0.log"
Sep 17 18:51:30.549091 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.549071    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kp8xn_3e13d86a-6c05-4f96-a87a-a075c01901ac/whereabouts-cni/0.log"
Sep 17 18:51:30.700490 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.700416    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qdh2w_589f88ca-b342-488a-9e88-4c60c80dd6bb/network-metrics-daemon/0.log"
Sep 17 18:51:30.721329 ip-10-0-141-122 kubenswrapper[2561]: I0917 18:51:30.721306    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qdh2w_589f88ca-b342-488a-9e88-4c60c80dd6bb/kube-rbac-proxy/0.log"